Apparatus, method and storage medium

ABSTRACT

In one embodiment of the present disclosure, an apparatus that generates album data including a plurality of pages has a page generation unit configured to generate common pages used in common in a plurality of pieces of album data and individual pages corresponding to a main object and a determination unit configured to determine, after sorting the common pages relating to generation-target album data based on an image capturing time representing the common pages, positions of the individual pages relating to the generation-target album data.

BACKGROUND OF THE INVENTION Field of the Invention

The present disclosure relates to an apparatus, a method, and a storagemedium and more specifically, to an apparatus, a method, and a storagemedium for creating an album.

Description of the Related Art

In recent years, the number of photos captured by a user is increasingrapidly due to the spread of smart devices and improvement of the cameraperformance of smart devices, in addition to the spread of digitalcameras. Accompanying an increase in the number of captured photos, thenumber of variations of captured objects is also increasing.

Conventionally, a method for creating an album by laying out photoscaptured by a digital camera is known. Japanese Patent Laid-Open No.2016-66328 has proposed a method for creating an album by a plurality ofusers collaborating.

SUMMARY OF THE INVENTION

In a case where an album is created by a plurality of users providingphotos, it is necessary to create an album by taking into considerationphotos each user desires to use. With the technique disclosed inJapanese Patent Laid-Open No. 2016-66328, it is possible to create analbum by each user participating in album creation. However, thetechnique disclosed in Japanese Patent Laid-Open No. 2016-66328 has nottaken into consideration an album having both contents common to allusers and contents customized to each individual.

Consequently, in view of the above-described problem, the purpose of thepresent disclosure is to create album data having both the contentscommon to a plurality of pieces of album data and the contents inaccordance with each individual (object).

In one embodiment of the present disclosure, an apparatus that generatesalbum data including a plurality of pages has a page generation unitconfigured to generate common pages used in common in a plurality ofpieces of album data and individual pages corresponding to a main objectand a determination unit configured to determine, after sorting thecommon pages relating to generation-target album data based on an imagecapturing time representing the common pages, positions of theindividual pages relating to the generation-target album data.

Further features of the present disclosure will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a hardware configuration of an imageprocessing apparatus in a first embodiment;

FIG. 2A is a block diagram showing a software configuration of the imageprocessing apparatus in the first embodiment;

FIG. 2B is a detailed block diagram of a double-page spread creationunit in the first embodiment;

FIG. 3 is a diagram showing the relationship of FIG. 3A and FIG. 3B;

FIG. 3 A and FIG. 3B are each a diagram explaining layouts of images inalbums that are created in the first embodiment;

FIG. 4 is a diagram showing a GUI screen of an album creationapplication in the first embodiment;

FIG. 5A is a flowchart of automatic layout processing in the firstembodiment;

FIG. 5B is a flowchart of image analysis processing in the firstembodiment;

FIG. 5C is a flowchart of double-page spread creation processing in thefirst embodiment;

FIG. 6 is a diagram explaining an image group division in the firstembodiment;

FIG. 7 is a diagram explaining scene classification in the firstembodiment;

FIG. 8 is a diagram explaining image selection in the first embodiment;

FIG. 9 is a block diagram explaining template determination in the firstembodiment;

FIG. 10 is a flowchart of template determination processing in the firstembodiment;

FIG. 11 is a diagram explaining an image layout determination method inthe first embodiment;

FIG. 12 is a detailed block diagram of a double-page spread combinationunit in the first embodiment;

FIG. 13 is a flowchart of double-page spread combination processing inthe first embodiment;

FIG. 14A is a block diagram showing a software configuration of an imageprocessing apparatus in a second embodiment;

FIG. 14B is a detailed block diagram of a double-page spread creationunit in the second embodiment;

FIG. 15A is a flowchart of automatic layout processing in the secondembodiment;

FIG. 15B is a flowchart of image analysis processing in the secondembodiment;

FIG. 15C is a flowchart of double-page spread creation processing in thesecond embodiment;

FIG. 16 is a detailed block diagram of a double-page spread combinationunit in the second embodiment;

FIG. 17 is a flowchart of double-page spread combination processing inthe second embodiment;

FIG. 18A is a block diagram showing a software configuration of an imageprocessing apparatus in a third embodiment;

FIG. 18B is a detailed block diagram of a double-page spread creationunit in the third embodiment;

FIG. 19A is a flowchart of automatic layout processing in the thirdembodiment;

FIG. 19B is a flowchart of image analysis processing in the thirdembodiment;

FIG. 19C is a flowchart of double-page spread creation processing in thethird embodiment;

FIG. 20 is a flowchart of number of double-page spreads calculationprocessing in the third embodiment;

FIG. 21A is a block diagram showing a software configuration of an imageprocessing apparatus in a fourth embodiment;

FIG. 21B is a detailed block diagram of a double-page spread creationunit in the fourth embodiment;

FIG. 22A is a flowchart of automatic layout processing in the fourthembodiment;

FIG. 22B is a flowchart of image analysis processing in the fourthembodiment;

FIG. 22C is a flowchart of double-page spread creation processing in thefourth embodiment;

FIG. 23 is a detailed block diagram of a double-page spread combinationunit in the fourth embodiment;

FIG. 24 is a flowchart of double-page spread combination processing inthe fourth embodiment;

FIG. 25A is a block diagram showing a software configuration of an imageprocessing apparatus in a fifth embodiment;

FIG. 25B is a detailed block diagram of a double-page spread creationunit in the fifth embodiment;

FIG. 26A is a flowchart of automatic layout processing in the fifthembodiment;

FIG. 26B is a flowchart of image analysis processing in the fifthembodiment;

FIG. 26C is a flowchart of double-page spread creation processing in thefifth embodiment;

FIG. 27 is a detailed block diagram of a double-page spread combinationunit in the fifth embodiment;

FIG. 28 is a flowchart of double-page spread combination processing inthe fifth embodiment;

FIG. 29 is a diagram for explaining effects of the first embodiment;

FIG. 30A and FIG. 30B are each a diagram for explaining effects of thesecond embodiment;

FIG. 31A and FIG. 31B are each a diagram for explaining effects of thethird embodiment;

FIG. 32 is a diagram for explaining effects of the fourth embodiment;

FIG. 33 is a diagram for explaining effects of the fifth embodiment;

FIG. 34A is a block diagram showing a software configuration of an imageprocessing apparatus in a sixth embodiment;

FIG. 34B is a detailed block diagram of a double-page spread creationunit in the sixth embodiment;

FIG. 35A is a flowchart of automatic layout processing in the sixthembodiment;

FIG. 35B is a flowchart of image analysis processing in the sixthembodiment;

FIG. 35C is a flowchart of double-page spread creation processing in thesixth embodiment;

FIG. 36A is a block diagram showing a software configuration of an imageprocessing apparatus in a seventh embodiment;

FIG. 36B is a detailed block diagram of a double-page spread creationunit in the seventh embodiment;

FIG. 37A is a flowchart of automatic layout processing in the seventhembodiment;

FIG. 37B is a flowchart of image analysis processing in the seventhembodiment;

FIG. 37C is a flowchart of double-page spread creation processing in theseventh embodiment;

FIG. 38 is a block diagram showing a software configuration of an imageprocessing apparatus in an eighth embodiment;

FIG. 39 is a flowchart of automatic layout processing in the eighthembodiment; and

FIG. 40 is a diagram for explaining effects of the eighth embodiment.

DESCRIPTION OF THE EMBODIMENTS

In the following, embodiments of the present disclosure are explainedwith reference to the drawings. The following embodiments are notintended to limit the present disclosure and all combinations offeatures explained in the present embodiments are not necessarilyindispensable to the present disclosure. Explanation is given byattaching the same symbol to the same configuration.

First Embodiment

In the present embodiment, in an image processing apparatus, an albumcreation application (hereinafter, also referred to as “application”)for creating a photo album (hereinafter, also referred to simply as“album”) is caused to run. Then, on the application, a plurality ofpieces of album data in which a plurality of photo images is laid outautomatically is created. In the present embodiment, a double-pagespread common to at least two albums and an individual double-pagespread in each album are created.

The total number of double-page spreads configuring each album is thesame in all the albums in accordance with user specification. However,each of the number of common double-page spreads and the number ofindividual double-page spreads for each album (that is, for each user)changes in accordance with the image group and the setting. Then, in thepresent embodiment, the double-page spreads are arranged in order of theimage capturing time.

<Hardware Configuration of Image Processing Apparatus>

In the following, the hardware configuration of an image processingapparatus in the present embodiment is explained by using FIG. 1. In thefollowing, a case is explained where an image processing apparatus is aninformation processing apparatus (PC), but it may also be possible toadopt another apparatus, such as a smartphone, as an image processingapparatus.

As shown in FIG. 1, an image processing apparatus 100 has a CPU 101, aROM 102, a RAM 103, an HDD 104, a display 105, a keyboard 106, a mouse107, and a data communication device 108. These units are connected by adata bus 109 and capable of performing transmission and reception ofdata with one another.

The CPU 101 controls the entire image processing apparatus 100. Further,the CPU 101 performs an image processing method that is explained in thepresent embodiment in accordance with a program. In FIG. 1, the imageprocessing apparatus has one CPU, but the image processing apparatus mayhave a plurality of CPUs.

In the ROM 102, programs that are executed by the CPU 101 are stored.The RAM 103 provides a memory for temporarily storing various kinds ofinformation at the time of execution of programs by the CPU 101. In theHDD 104, a database or the like storing image files and processingresults of an image analysis and the like is stored and in the presentembodiment, in this HDD 104, the application program for album creationis stored. This application program will be described later by usingFIG. 4 and the like.

The display 105 is a device for presenting a user interface(hereinafter, UI) of the present embodiment and layout results of imagesto a user by displaying them. The display 105 may have a touch sensorfunction. The keyboard 106 is one of input devices possessed by theimage processing apparatus 100 and for example, used to inputpredetermined information onto the GUI displayed on the display 105. Inthe present embodiment, a user inputs the number of double-page spreadsof an album via the keyboard 106. The mouse 107 is one of the inputdevices possessed by the image processing apparatus 100 and for example,used to click and press down a button on the GUI displayed on thedisplay 105.

The data communication device 108 is a device for communicating with anexternal device, such as a printer and a server. For example, the dataof the automatic layout is transmitted to a printer or a serverconnected to the image processing apparatus 100 via the datacommunication device 108. The data bus 109 connects each of thecomponents described above and the CPU 101. The above is the contents ofthe hardware configuration of the image processing apparatus in thepresent embodiment.

<Software Configuration of Image Processing Apparatus>

In the following, the software configuration of the image processingapparatus in the present embodiment, in other words, the functionconfiguration implemented by the album creation application installed inthe image processing apparatus is explained by using FIG. 2A. The albumcreation application boots up by the icon of the application displayedon the display 105 being double-clicked by a user using the mouse 107.Although the album creation application has a variety of functions, inthe following, the automatic layout function provided by an automaticlayout processing unit 202 is explained mainly, which is a particularlyimportant function in the present embodiment. It is assumed that thealbum creation application is installed in advance in the imageprocessing apparatus 100. However, the album creation application of thepresent embodiment is not limited to that installed in advance in theimage processing apparatus 100 and may a Web application that isperformed on a Web browser of the image processing apparatus 100. In acase of the Web application, the program is read by accessing apredetermined Web site.

As shown in FIG. 2A, this application has an album creation conditionsetting unit 201, the automatic layout processing unit 202, and an albumdisplay unit 213. The “automatic layout function” is a function ofcreating album data by laying out an image of a captured photo afterclassifying the image according to contents and attribute thereof andthen selecting the image. The created album data is displayed on thedisplay 105. It is preferable for the created album data to be uploadedto an external server via a predetermined network (internet or thelike). Then, in the external server, print data is generated based onthe album data and a book-bound album made of paper media is completedby a printing apparatus performing printing and bookbinding based on theprint data.

The album creation condition setting unit 201 sets album creationconditions in accordance with the mouse operation of a user to theautomatic layout processing unit 202. In the present embodiment, asalbum creation conditions, the image group used for the album, the totalnumber of double-page spreads per album, the commodity material of thealbum, the number of albums, and the main object for each album are set.It may also be possible to set the image group by using appendedinformation and attribute information on an individual piece of imagedata, for example, such as the image capturing date, or to set the imagegroup based on the file system structure in which image data is saved,such as specification of the device and the directory. The “double-pagespread” corresponds to, for example, one display window in display, andcorresponds to a pair of pages (that is, two pages) adjacent to eachother printed on different sheets in printing. In the presentembodiment, the “double-page spread” is also called a “spread page”.Further, in a case of a simple representation as a “page”, it is assumedthat both the concept of the “double-page spread (that is, two pages)”such as this and the concept of the “one page” are included.

Further, the main object is an object that is laid out in a particularlyenhanced manner in an album. For example, mention is made of a specificperson, a category of object, such as dog, cat, flower, and dish, a hueof an object and the entire image, and the like. Here, for explanation,a person is taken as a main object.

Here, albums that are created in the present embodiment are explained byusing FIG. 3A and FIG. 3B. In the present embodiment, a plurality ofdifferent variations of album is created based on the number of albumsand the main object for each album that are set by the album creationcondition setting unit 201. In the example in FIG. 3A and FIG. 3B, album1, album 2, and album 3 are created as a plurality of differentvariations of album. The album 1 is an album created by focusingattention on a main object A, the album 2 is an album created byfocusing attention on a main object B, and the album 3 is an album byfocusing attention on a main object C.

Each album includes a “common double-page spread” on which the sameimages are laid out in the same arrangement in all the plurality ofdifferent variations of album and an “individual double-page spread” onwhich images different from those of another album are laid out orimages are laid out in different arrangement. In other words, the“common double-page spread” is a spread page whose layout is made incommon to all albums created for each object of interest. That is, the“common double-page spread” refers to the spread page whose layout ismade in common to the album 1, the album 2, and the album 3. On theother hand, the “individual double-page spread” is a spread page whoselayout is made differently in each album created for each object ofinterest. That is, the “individual double-page spread” is a spread pagewhose layout is made differently in the album 1, the album 2, and thealbum 3 and for example, the individual double-page spread provided inthe album 1 is not included in the albums 2 and 3. In the example inFIG. 3A and FIG. 3B, each of the album 1 to the album 3 includes acommon double-page spread 302. On the other hand, the album 1 includesan individual double-page spread 303, the album 2 includes an individualdouble-page spread 304, and the album 3 includes an individualdouble-page spread 305.

An image 301 is an image arranged on the double-page spread and thecharacters in the image 301 indicate the main object captured in theimage 301. As shown in FIG. 3A and FIG. 3B, the common double-pagespread 302 is laid out so that an image in which a plurality of mainobjects is captured is included. On the other hand, each of theindividual double-page spreads 303 to 305 is laid out so that an imagein which a specific main object is captured is included, such as thatthe individual double-page spread 303 includes an image in which themain object A is captured, and so on. As described above, in a casewhere a plurality of albums is created in the present embodiment, eachof the plurality of albums includes a double-page spread created basedon a different reference. The number of albums created by focusingattention on each main object is not limited to one. That is, the albumcreated by focusing attention on the main object A may be created forthe A's family, not only for A corresponding to the main object A.

An image acquisition unit 203 acquires an image group that satisfies thealbum creation conditions set by the album creation condition settingunit 201 from among the images saved in the HDD 104. The image groupreferred to here is the image group that is the layout candidate at thetime of album creation. For example, in a case where the period fromJanuary 1, XXXX to December 31, XXXX is specified as the image capturingdate, all the images captured during the period from January 1, XXXX toDecember 31, XXXX correspond to the image group of the layout candidate.As the images saved in the HDD 104, mention is made of still images andcutout images cut out from a moving image. The still image and thecutout image are images acquired by an image capturing device, such as adigital camera and a smart device. The image capturing device may becomprised by the image processing apparatus 100 or comprised by anexternal device of the image processing apparatus 100. In a case wherethe image capturing device is the external device, the image acquisitionunit 203 acquires an image via the data communication device 108.Further, the still image and the cutout image may be images acquiredfrom an external network or server via the data communication device108. As the image acquired from a network or server, mention is made ofa social networking service image (hereinafter, SNS image). The CPU 101analyzes data attached to image data for each image by executing theprogram of the OS and finds the source in which the image is saved, thatis, the source from which the image is acquired. However, it may also bepossible to manage the source from which the image is acquired withinthe application by acquiring the image from SNS via the application. Theimage that is acquired by the image acquisition unit 203 is not limitedto those described above and may be another kind of image.

An image analysis unit 204 analyzes the image data acquired by the imageacquisition unit 203. In the present embodiment, the image analysis unit204 derives the feature quantity of the image and performs objectdetermination within the image, face detection, expression recognitionof the detected face, and individual recognition of the detected face.Further, the image analysis unit 204 acquires information on the imagecapturing date by referring to data (for example, Exif information)attached to the image data acquired from the HDD 104. The informationobtained as the results of the image analysis unit 204 analyzing theimage data is called “analysis information”.

A main object setting unit 205 sets a main object (protagonist) that isenhanced for each album. In the present embodiment, the main objectsetting unit 205 acquires information on the main object that isenhanced for each album, which is sent from the album creation conditionsetting unit 201.

An image scoring unit 208 scores each image based on the album creationconditions set by the album creation condition setting unit 201 and theanalysis information derived by the image analysis unit 204. In thepresent embodiment, each image is scored so that an image suitable tothe layout has a high score. At the time of performing scoring so thatan image suitable to the layout has a high score, the image analysisinformation, image classification results, image information, and thealbum creation conditions are used. However, another piece ofinformation may be used additionally or alternatively. As the imagesuitable to the layout, mention is made of, for example, an image whoseaesthetic appreciation is high, such as that contrast is high and theedge is sharp, and an image in which the object that is the subject(so-called main object) is captured in a large size, such as a person,an animal, and a building. Further, the image scoring unit 208 alsoperforms scoring for each main object that is set by the main objectsetting unit 205. For example, the image in which the set main object isnot captured is scored low even though the image has a very highaesthetic appreciation. On the other hand, the image in which the setmain object is captured in a large size is scored high even though theimage is an image whose image capturing state not good, such as theimage captured against the light.

A scene division unit 206 divides the image group acquired by the imageacquisition unit 203 according to scene by using the analysisinformation derived by the image analysis unit 204. The “scene” referredto here means a group (sub image group) of images grouped in accordancewith analysis information. For example, as regards two images whoseimage capturing times are different, the scene division unit 206determines that the two images are the same scene in a case thedifference between the image capturing times is less than or equal to apredetermined threshold value or determines that the images in which thecaptured person and background are the same as the same scene.

A scene classification unit 207 determines a relevant category for eachscene obtained by the scene division unit 206. As the category of ascene, mention is made of, for example, a situation, such as indoors andoutdoors, an event at the time of capturing photos, such as a journeyand a wedding.

A double-page spread creation parameter generation unit 209 generates aplurality of parameters for creating a double-page spread based on themain object that is set by the main object setting unit 205 and theanalysis information derived by the image analysis unit 204. Theparameters referred to here are setting values that determine thedouble-page spread creation reference. As the parameters such as those,mention is made of, for example, weights and threshold values thatdetermine priority of scores obtained by the image scoring unit 208,which is used at the time of selecting images that are used for thedouble-page spread. The parameters are setting values to select whetherto give priority to an image in which only the specific main object iscaptured or give priority to an image in which two or more main subjectsare captured among the images in which the main objects set by the mainobject setting unit 205 are captured. By using a plurality ofparameters, the creation reference changes depending on the double-pagespread of each album. That is, the double-page spread creation referenceis a reference of page creation of each piece of album data.

A double-page spread creation unit 210 creates a double-page spread byselecting and arranging images that are used for the double-page spreadin accordance with the parameters generated by the double-page spreadcreation parameter generation unit 209. That is, the double-page spreadcreation unit 210 functions as an album data page generation unit.

A double-page spread combination unit 211 combines the double-pagespreads created by the double-page spread creation unit 210.Specifically, the double-page spread combination unit 211 creates thelayout of the entire album by determining the order number of eachdouble-page spread created by the double-page spread creation unit 210.That is, the double-page spread combination unit 211 functions as a datageneration unit configured to generate album data in which each page isarranged.

A layout information output unit 212 outputs layout information inaccordance with the layout of the entire album, which is created by thedouble-page spread combination unit 211. The layout information is, forexample, image data in the bitmap format in which each image is arrangedin accordance with a predetermined layout.

The album display unit 213 displays images of the created album on thedisplay 105 based on the layout information that is output by the layoutinformation output unit 212.

In a case where the program of the album creation application in thepresent embodiment is installed in the image processing apparatus 100,the boot icon of the application is displayed on the top screen(desktop) of the OS (Operating System) that runs on the image processingapparatus 100. In a case where a user double-clicks the boot icon on thedesktop displayed on the display 105 with the mouse 107, the program ofthe album creation application saved in the HDD 104 is loaded onto theRAM 103. Then, the program loaded onto the RAM 103 is executed by theCPU 101 and the album creation application boots up. The above is thecontents of the software configuration of the image processing apparatusin the present embodiment.

<GUI Screen>

In the following, the GUI screen of the album creation application inthe present embodiment is explained by using FIG. 4. FIG. 4 is a diagramshowing a GUI screen 401 that is provided by the album creationapplication having booted up and displayed on the display 105. It ispossible for a user to set the album creation conditions via the GUIscreen 401.

The GUI screen 401 has a path box 402 and a folder selection button 403as a setting unit of photo images that are included in an album. Thepath box 402 is a box for indicating the saving location (path) in theHDD 104 of an image group that is the target of album creation. Thefolder selection button 403 is a button for selecting the folderincluding the image group that is the target of album creation. In acase where a user clicks the folder selection button 403 with the mouse107, a tree configured by a plurality of folders is displayed. Then, ina case where a user selects the folder including the image group that isthe target of album creation, the folder path of the selected folder isdisplayed in the path box 402.

A number of double-page spreads box 404 is a box for specifying thetotal number of double-page spreads in one album. It is possible for auser to directly input a figure(s) in the number of double-page spreadsbox 404 by using the keyboard 106 or input a figure(s) in the number ofdouble-page spreads box 404 from a list by using the mouse 107. A numberof albums box 410 is a box for specifying the number of albums to becreated.

A main object specification unit 405 is an element for specifying a mainobject that is enhanced for each of the albums corresponding to thenumber that is input to the number of albums box 410. For example, it ispossible for a user to specify a main object by dragging and dropping animage in which the main object is captured in the main objectspecification unit 405 by using the mouse 107.

A number of double-page spread photos setting unit 406 is a slider barfor setting the number of images that are arranged on each double-pagespread of the album to be created. In a case where the slider is movedto the “large” side, the number of images that are arranged on eachdouble-page spread increases. On the other hand, in a case where theslider is moved to the “small” side, the number of images that arearranged on each double-page spread decreases.

A commodity material specification unit 407 is an element for settingthe commodity material by which the album is created. As specific itemsrelating to the commodity material that is set, it is possible to adoptthe size of the album and the kind of sheet of the album. It may also bepossible to set the kind of front cover and the kind of binding portion.

An OK button 408 is a button for determining selected conditions as thealbum creation conditions. In a case where a user clicks the OK button408 with the mouse 107, the album creation conditions are settled andthe album creation conditions are conveyed to the automatic layoutprocessing unit 202 via the album creation condition setting unit 201.To explain specifically, the path information that is input to the pathbox 402 is conveyed to the image acquisition unit 203. Further, thevalue of the total number of double-page spreads that is input to thenumber of double-page spreads box 404 is conveyed to the double-pagespread creation unit 210. The value of the number of albums that isinput to the number of albums box 410 and the information on the mainobject specified in the main object specification unit 405 are conveyedto the double-page spread creation parameter generation unit 209.

A Reset button 409 is a button for resetting each of the set contents onthe display screen. The above is the contents of the GUI screen of thealbum creation application in the present embodiment.

<Automatic Layout Processing>

In the following, the automatic layout processing in the presentembodiment is explained by using FIG. 5A to FIG. 5C. FIG. 5A to FIG. 5Care each a flowchart of processing to perform the automatic layout ofthe album creation application according to the present embodiment. Theflowcharts shown in FIG. 5A to FIG. 5C are implemented by, for example,the CPU 101 reading the programs stored in the HDD 104 onto the ROM 102or the RAM 103 and executing the programs.

At step S501, the album creation condition setting unit 201 sets albumcreation conditions. In the following, “step S-” is simply abbreviatedto “S-”. In the present embodiment, as the album creation conditions,the image group that is used in the album, the number of double-pagespreads, the number of albums, the main object of each album, themagnitude of the number of images that are arranged on each double-pagespread, and the commodity material by which the album is created are set(see FIG. 4).

At S502, the image acquisition unit 203 reads images that satisfy theconditions set at S501 relating to the image group that is used in thealbum from the HDD 104. Then, the image acquisition unit 203 loads theread images onto the RAM 103.

At S503, the image analysis unit 204 analyzes the images loaded onto theRAM 103 at S502. Here, the image analysis at this step is explained byusing FIG. 5B.

At S50301, the image analysis unit 204 acquires the information on theimage capturing date corresponding to the image data acquired by theimage acquisition unit 203. In the present embodiment, the informationon the image capturing date is acquired based on the Exif informationattached to each piece of image data.

At S50302, the image analysis unit 204 derives the feature quantity ofimage quality for the image data acquired by the image acquisition unit203. As the feature quantity of image quality, for example, there is anin-focus degree. As a method of determining the in-focus degree, it ispossible to use an edge detection method, and as the edge detectionmethod, it is possible to use a publicly known Sobel filter. Bydetecting an edge of the image with the Sobel filter and dividing thedifference in luminance between the starting point and the endpoint ofthe edge by the distance between the starting point and the endpoint,the luminance gradient, that is, the slope of the edge is calculated. Bycalculating the average slope of the edge in the image, it is possibleto regard the image whose average slope is large as being in focuscompared to the image whose average slope is small. In the presentembodiment, whether the in-focus degree is acceptable is determined bysetting a plurality of threshold values for measuring the magnitude ofthe calculated average slope of the edge and determining whether thecalculated slope of the edge is larger than or equal to one of thethreshold values. Specifically, as two different slope threshold values,a first slope threshold value and a second slope threshold value (firstslope threshold value>second slope threshold value) are set and thein-focus degree is determined with three levels of ∘, Δ, and x. In acase where the average slope of the edge in the image is larger than orequal to the first threshold value, the in-focus degree is determined tobe favorable (indicated by ∘), Further, in a case where the averageslope of the edge in the image is less than the first threshold valueand larger than or equal to the second threshold value, the in-focusdegree is determined to be acceptable (indicated by Δ) and in a casewhere the average slope of the edge in the image is less than the secondthreshold value, the in-focus degree is determined to be unacceptable(indicated by x).

At S50303, the image analysis unit 204 performs object detection andclassification of the detected object for the image data acquired by theimage acquisition unit 203. In the present embodiment, the face isdetected as the object. As the face detection method, it is possible toadopt any publicly known method and as the publicly known method such asthis, for example, there is AdaBoost that creates a strong discriminatorfrom a plurality of prepared weak discriminators. In the presentembodiment, by using the strong discriminator created by AdaBoost, theface of a person is detected. At S50303, the face is detected and at thesame time, the top-left coordinate values and the bottom-rightcoordinate values of the area of the detected face in the image areacquired. By finding these two kinds of coordinate value, it is possibleto specify the position of the face and the size of the face. Further,similar to the face of a person, by performing AdaBoost to detect eachof an animal, such as a dog and a cat, and a dish, it is possible todetect the objects of a person, an animal, and a dish and at the sametime, classify the object within the image. The detection-target objectis not limited to those described above and the object may be a flower,a building, an ornament, and the like. Further, here, the case isexplained where objects are classified by using AdaBoost, but it mayalso be possible to adopt image recognition by a deep neural network.That is, it may also be possible to generate a learned model foroutputting the kind of object by taking the image as input data and thekind of object included in this image as training data and perform imagerecognition by using the learned model. At the time of learning, it isrecommended to cause the model to learn by associating a plurality ofpieces of input data with training data and using the algorithm of thedeep neural network.

At S50304, the image analysis unit 204 performs individual recognitionfor the face detected by the image analysis unit 204. First, the imageanalysis unit 204 derives a degree of similarity between the extractedface image and the representative face image saved in a face dictionarydatabase for each individual ID. Then, the individual ID whose deriveddegree of similarity is higher than or equal to a predeterminedthreshold value and whose degree of similarity is the highest isdetermined to be ID corresponding to the extracted face image. That is,the person corresponding to the individual ID whose degree of similarityis higher than or equal to the predetermined threshold value and whosedegree of similarity is the highest is specified as the person of theextracted face image. In a case where all the degrees of similarityderived for each individual ID are less than the predetermined thresholdvalue, the person of the extracted face image is regarded as a newperson and a new individual ID is allocated to the person and the personis registered to the face dictionary database. The face dictionarydatabase is stored in, for example, the HDD 104.

Explanation is returned to FIG. 5A. At S504, the image analysis unit 204determines whether the processing at S502 to S503 has been completed forall the images of the image group that satisfies the conditions set atS501. In a case where determination results at this step areaffirmative, the processing advances to S505 and on the other hand, in acase where the determination results are negative, the processingreturns to S502.

At S505, the scene division unit 206 performs a scene division for theimage group acquired by the image acquisition unit 203. The scenedivision refers to dividing the acquired image group into a plurality ofsub image groups. Specifically, by using the information on the imagecapturing date, which has already been acquired at S50301, the imagegroup is divided into a plurality of sub image groups based on the timedifference in the image capturing date between images. An example of anactual division is as follows.

First, attention is focused on the image whose image capturing time isthe oldest (or the newest) and the time difference against the imagewhose image capturing time is the second oldest (or the second newest)is calculated, and then whether the calculated time difference is largerthan or equal to a predetermined threshold value is determined. Theprocessing such as this is performed for all the images by sequentiallyreplacing the image of interest with the image whose image capturingtime is newer (or older). “Division” in the present embodiment meansdividing the image group into an image group whose image capturing timeis new and an image group whose image capturing time is old at aboundary between two images. In the present embodiment, in a case wherethe difference between the image capturing time of the image of interestand the image capturing time of the image that is next older (or newer)is 16 hours or more, the image group is divided so that these imagesbelong to different sub image groups. The threshold value of the timedifference used at the time of division is not limited to this. FIG. 6shows the results of dividing the image group by using the scenedivision method described above.

Further, in the present embodiment, the scene division is performed byusing the information on the image capturing date, but the informationthat is used is not limited to this. For example, it may also bepossible to divide the image group by using the information on the imagecapturing location so that images whose image capturing locations areclose belong to the same scene. As another example, it may also bepossible to use the individual recognition results. For example, in agraduation album, by registering in advance the image of a studentbelonging to a certain group (class, club activity, and the like) to theapplication, it may also be possible to take an image group includingthe images in which the student belonging the group is captured as onescene (sub image group). As still another example, information otherthan image information may be used. Here, a method of putting togethersimilar events as one scene is explained. The folder name in whichimages are saved and tag information attached in social networkingservice or the like are acquired as meta information accompanying theimage. For example, an image group having meta information including asearch word is taken as one scene by a search word, such as “athleticmeet” and “school excursion”. As regards specification of a search word,it may also be possible for a user to select a search word from searchwords incorporated in advance in the application via the album creationcondition setting unit 201 or input a search word to a text box.

At S506, the scene classification unit 207 performs the sceneclassification for each sub image group obtained at S505. In thefollowing, explanation is given by taking a case as an example whereeach sub image group (each scene) is classified into one of threecategories, that is, “journey”, “daily life”, and “ceremony”, but itemsaccording to which classification is performed are not limited to those.

First, a plurality of sub image groups having been determined in advanceas to into which of journey, daily life, and ceremony the sub imagegroup is classified is acquired and the feature quantity of imagecapturing for each of the acquired plurality of sub image groups isacquired. The feature quantity of image capturing that is acquired hereis, for example, the image capturing period, the number of capturedimages, and the number of captured persons. The image capturing periodis the time difference between the image capturing time of the oldestimage and the image capturing time of the newest image, both imagesbeing included in the sub image group. The number of captured images isthe number of images (that is, the number of photos) included in the subimage group. The number of captured persons is the number of faces inthe image in which a face or faces are captured, that is, the number offaces included in one image. Then, for the plurality of sub imagegroups, the average value and the standard deviation of the imagecapturing period, the average value and the standard deviation of thenumber of captured images, and the average value and the standarddeviation of the number of persons per image are found. In the presentembodiment, it is assumed that the number of faces per image is thenumber of persons per image.

FIG. 7 shows an example of the average value and the standard deviationof the image capturing period (time), the average value and the standarddeviation of the number of captured images, and the average value andthe standard deviation of the number of persons per image, which arefound for the plurality of sub image groups. These found values areincorporated in advance in the program of the album creationapplication. That is, in the design stage of the application, parametersare generated by learning using images collected in advance and thegenerated parameters are incorporated in the program. After the albumcreation application is booted up, the average value of each of theimage capturing period, the number of captured images, and the number ofpersons per image is calculated for each sub image group obtained as aresult of the division at S505 for the image group specified by a uservia the path box 402. Then, for each scene, a score is calculated inaccordance with an equation below by using the parameters (specifically,the average value and the standard deviation of each feature quantity ofthe image capturing period, the number of captured images, and thenumber of captured persons of each sub image group) incorporated inadvance in the application described previously.score for each scene and for each feature quantity=50−|10×(average valuefor each scene and for each feature quantity−feature quantity for eachsub image group)/standard deviation for each scene and for each featurequantity|  (1)average score for each scene=(score of image capturing period for eachscene+score of number of captured images for each scene+score of numberof captured persons for each scene)/number of items of featurequantity  (2)

By equation (1), the score of each scene and each feature quantity ofthe sub image group of interest is found. For example, for the journeyscene, the score for each of the image capturing period, the number ofcaptured images, and the number of captured persons is found. Thesescores are averaged by equation (2) and the average is taken as thescore for the journey scene. Similarly, the scores are also obtained forthe other scenes, that is, daily life and ceremony. By the methodexplained above, the average score of the journey scene, the averagescore of the daily life scene, and the average score of the ceremonyscene are calculated for each sub image group. The number of items ofthe feature quantity in equation (2) is three.

The scene having the highest score of the scores for each scenecalculated for each sub image group is classified as the scene of thesub image group. In a case where two or more scenes have the same score,the scene classification is performed in accordance with predeterminedpriority. For example, in the present embodiment, it is determined inadvance that priority is in order of daily life, ceremony, and journeyand the highest priority is given to daily life. The order of priorityis not limited to this and it may also be possible to enable a user tochange the order of priority.

Sub image groups (1) to (8) in FIG. 6 show sub image groups obtained bythe scene division for the image group. As regards the sub image group(5) of these sub image groups, it is assumed that the image capturingperiod is 36 hours, the number of captured images is 300, and the numberof captured persons is 1.7. For those, the average score of journey is45.32, the average score of daily life is 18.38, and the average scoreof ceremony is −29.92 and as a result of this, the scene of the subimage group (5) is classified as journey. The classified sub image groupis managed in association with a scene ID so that it is possible toidentify a scene. In the present embodiment, the case is explained wherethe image capturing period, the number of captured images, and thenumber of captured persons are adopted as the feature quantities ofimage capturing, but the feature quantities of image capturing are notlimited to those. For example, it may also be possible to adopt a flagas the feature quantity, which indicates whether or not the sky iscaptured by image segmentation using machine learning in order tospecify indoor or outdoor. Alternatively, it may also be possible tofurther classify ceremony into smaller ones. For example, in a casewhere ceremony is classified into one of wedding and reception, the twopersons most frequently captured in all the images are taken as a groomand a bride by using individual recognition results. Then, it may alsobe possible to classify a scene in which there are many photos in whichonly the groom and the bride are captured into wedding and a scene inwhich there are many photos in which persons other than the groom andthe bride are also captured into reception. Further, it may also bepossible to derive the feature quantity by machine learning. Forexample, it may also be possible to perform the scene classification bypreparing the scene desired to be classified and the image grouprepresenting the scene and performing learning by using ConvolutionNeural Network (CNN) so that the input is the image and the output isthe scene classification results. That is, it may also be possible togenerate a learned model for outputting a scene by taking an image asinput data and the scene of the image as training data and perform imagerecognition by using the learned model. At the time of learning, it isrecommended to cause the model to learn by associating a plurality ofpieces of input data with training data and using the algorithm of CNN.

At S507, the main object setting unit 205 sets the main object of eachalbum. In the present embodiment, the information on the main object ofeach album is acquired, which is transmitted from the album creationcondition setting unit 201. Here, for explanation, it is assumed thatthe main object A is set to the album 1 and the main object B is set tothe album 2 for the two kinds of album.

At S508, the image scoring unit 208 scores each image. This processingis called image scoring processing. The “score” that is derived at thisstep is a parameter for evaluating use applicability to an album and ascore obtained by evaluation from the perspective, to be describedlater, is given for each piece of image data and the score is referredto at the time of image selection, to be described later. In the presentembodiment, in addition to the level of image quality, a score is givenfor each main object set at S507. First, a score in accordance with thefeature quantity of image quality derived at S50302 is given. Here, itis assumed that 1 is given in a case where the in-focus degree is ∘(favorable), 0.5 is given in a case of Δ (acceptable), and 0 is given ina case of x (unacceptable). Due to this, it is possible to increase thescore of an image in focus. In the present embodiment, as the featurequantity of image quality, the in-focus degree is adopted, but thefeature quantity of image quality is not limited to this. For example,it may also be possible to use the image size, the image capturinginformation, such as information on the lens used at the time of imagecapturing, or the compression format of the image that is input to theapplication.

After giving a score in accordance with the feature quantity of imagequality, the image scoring unit 208 gives a score for each main objectthat is set. That is, in the present embodiment, three kinds of scoreare given to one image, specifically, scores obtained by evaluation fromthree perspectives, that is, whether the image quality is good, whetherthe main object A is captured, and whether the main object B iscaptured, are given. For scoring of each main object, it is possible touse the size of the face derived at S50303 and the individualrecognition results at S50304. In a case where the main object is notcaptured in the image, 0 is given and in a case where the main object iscaptured, the ratio in which the face of the individual that is set asthe main object occupies the image size is given as a score of each mainobject. In the present embodiment, scoring for each main object sperformed by using the face size, but it may also be possible to use anitem other than the face size. For example, the expression of theperson, who is the main object, is determined and in a case where theface is a smiling face, the score may be increased. Further, in a casewhere an object other than a person is set as the main object, scoringis performed similarly in accordance with the object size by using theobject detection and the classification results at S50303.

At S509, the image scoring unit 208 determines whether the scoring atS508 has been completed for all the images of the image group acquiredby the image acquisition unit 203. In a case where determination resultsat this step are affirmative, the processing advances to S510 and on theother hand, in a case where the determination results are negative, theprocessing returns to S508.

At S510, the double-page spread creation parameter generation unit 209generates (sets) double-page spread creation parameters by receiving theinformation on the main object of each album from the main objectsetting unit 205 as well as receiving the album creation conditions fromthe album creation condition setting unit 201. The double-page spreadcreation parameters are setting values that are used at the time ofselecting and laying out images configuring the double-page spread, tobe described later. For example, the double-page spread creationparameter generation unit 209 generates a weight to control to which ofthe score in accordance with the image quality and the score of eachmain object priority is given at the time of selecting an image that isused for the double-page spread based on the score given to the image atS508, a threshold value that is the reference of image selection, andthe like.

The double-page spread creation parameter generation unit 209 generatesa “common parameter” common to all the albums and an “individualparameter” different for different albums as the double-page spreadcreation parameters. That is, the double-page spread creation parametergeneration unit 209 generates double-page spread creation parametersequal to (the number of albums+1). Due to this, it is possible to createa “common double-page spread” on which the double-page spread whoseimages configuring the double-page spread and whose arrangement are thesame appears in a plurality of albums and an “individual double-pagespread” on which the images configuring the double-page spread and thearrangement appear in a specific album. In the present embodiment, tothe common parameter, a weight is set by which each score given to theimage is evaluated equally. A score ^(I)score_(s) representing a certainimage I in a case where a common parameter w_(i) is used is expressed byequation (3) below.^(I)score_(S)=Σ_(i) w _(i)·^(I)score_(i)  (3)

Here, ^(I)score_(i) indicates the ith score given to the image I andw_(i) indicates the weight for each score. In the present embodiment, bysetting w_(i) to ⅓, the score of the image quality, the score of themain object A, and the score of the main object B are evaluated equally.Due to this, priority is given to the image whose image quality is goodand in which many persons whose score is high are captured.

On the other hand, to the individual parameter, a weight that givespriority to the image in which the main object of each album is capturedis set. A score ^(I)score_(I) representing the certain image I in a casewhere an individual parameter w_(o) is used is expressed by equation (4)below.

$\begin{matrix}{{{}_{}^{}{}_{}^{}} = \left\{ \begin{matrix}{0\mspace{284mu}} & {\left( {{{}_{}^{}{}_{}^{}} = 0} \right)} \\\frac{{w_{q} \cdot {{}_{}^{}{}_{}^{}}} + {w_{o} \cdot {{}_{}^{}{}_{}^{}}}}{2} & ({otherwise})\end{matrix} \right.} & (4)\end{matrix}$

Here, ^(I)score_(q) indicates the score of the image quality given tothe image I, ^(I)score_(o) indicates the score of a main object O givento the image I, w_(q) indicates the weight for the score of the imagequality, and w_(o) indicates the weight for the main object O. In thepresent embodiment, by setting w_(q) and w_(o) to 1 (w_(q)=w_(o)=1),only the image in which the main object is captured is evaluated and theimage to which priority is given is determined based on the imagequality and the size of the main object. In the present embodiment, asthe double-page spread creation parameter, the weight for the score isused, but the double-page spread creation parameter is not limited tothis. For example, it may also be possible to include the number ofdouble-page spreads to be created in the double-page spread creationparameters. Further, both ^(I)score_(s) and ^(I)score_(I) are found sothat the score is high in a case where the main object is captured, butthis is not limited. For the image in which the main object is notcaptured, such as a landscape, it is may also be possible to calculatethe score separately based on the score of the image quality, the imageinformation, and the like. For example, it may also be possible to takethe score of the image quality as the representative score for the imagewhose difference in the image capturing time from the image whose scoreis high is small. Due to this, it is also possible to give a high scoreto an image having a strong possibility of being related to the photo inwhich the main object is captured. Further, it is not necessary togenerate the double-page spread creation parameters each time. It mayalso be possible to set the common parameter w_(i) and the individualparameter w_(o) by respectively reading the value determined in advance.Further, it may also be possible to set the weight w_(o) for the mainobject O by reading the value determined in advance.

At S511, the double-page spread creation unit 210 creates a double-pagespread of the album by using the parameters generated at S510. Here, thedouble-page spread creation processing at this step is explained byusing FIG. 2B and FIG. 5C. FIG. 2B is a detailed block diagram of thedouble-page spread creation unit 210. FIG. 5C is a detailed flowchart ofdouble-page spread creation at S511.

At S51101, a double-page spread creation parameter setting unit 21001reads one of the parameters generated at S510. In the presentembodiment, the individual parameter of the albums 1, the individualparameter of the album 2, and the common parameter are read in thisorder.

At S51102, a double-page spread candidate image acquisition unit 21002acquires a candidate image that is used for the double-page spread fromamong all the image groups acquired by the image acquisition unit 203based on the score given at S508 and the double-page spread creationparameter set at S51101. In the present embodiment, based on the scorefound at S508 and the double-page spread creation parameter set atS51101, in accordance with equation (3) or equation (4), scoring foreach image is performed again (re-scoring of image). Then, the imagewhose score obtained as a result of re-scoring is higher than or equalto a predetermined threshold value is acquired as the candidate image.In a case where there is no candidate image, the image having thehighest score is taken as the candidate image. Here, the candidate imageis acquired from all the image groups, but this is not limited. It mayalso be possible to exclude the image having already been selected inimage selection processing, to be described later.

At S51103, a number of double-page spreads setting unit 21003 determineswhether the double-page spread creation parameter set at the immediatelyprevious S51101 is the individual parameter. In a case wheredetermination results at this step are affirmative, the processingadvances to S51104 and on the other hand, in a case where thedetermination results are negative, the processing advances to S51105.

In a case of YES at S51103 (that is, in a case where the individualparameter is set at the immediately previous S51101), at S51104, thenumber of double-page spreads setting unit 21003 sets the number ofdouble-page spreads based on the results of the scene division performedat S505 and the candidate image acquired at S51102. In the presentembodiment, in accordance with equation (5) and equation (6) below, thenumber of double-page spreads is set.number of double-page spreads for each scene=ceil(number of candidateimages in scene÷maximum number of images per double-page spread)  (5)number of individual double-page spreads=Σnumber of double-page spreadsfor each scene  (6)

By equation (5) and equation (6), the number of double-page spreadscreated with the individual parameter is found. The maximum number ofimages per double-page spread is set based on the user input via thenumber of double-page spread photos setting unit 406. In a case wherethe number of individual double-page spreads is larger than or equal tothe total number of double-page spreads that is set at S501, equation(7) below is followed.number of individual double-page spreads=total number of double-pagespreads−1  (7)

By equation (7), it is possible to cause double-page spreads createdwith different parameters, such as the individual parameter and thecommon parameter, to exist mixedly in one album. Due to this, for eachscene, it is possible to set the number of individual double-pagespreads in accordance with the number of images in which the main objectis captured frequently. It may also be possible to set an upper limitvalue and a lower limit value for the number of individual double-pagespreads.

On the other hand, in a case of NO at S51103 (that is, in a case wherethe common parameter is set at the immediately previous S51101), atS51105, the number of double-page spreads setting unit 21003 finds thenecessary number of double-page spreads as the number of commondouble-page spreads in accordance with equation (8) below.number of common double-page spreads=total number of double-pagespreads−MIN(total number of individual double-page spreads of eachalbum)  (8)

MIN ( ) is a function that returns the minimum value of the argument. Inthe present embodiment, the total number of double-page spreads of thealbum is made the same in each piece of album data, but there is apossibility that the number of double-page spreads of the individualdouble-page spread is different for each piece of album data.Consequently, it is possible to find the number of common double-pagespreads necessary for the album whose number of double-page spreads ofthe individual double-page spread is the minimum by equation (8). Forthe album whose number of double-page spreads of the individualdouble-page spread is the minimum, it is recommended to perform theprocessing so that the number of common double-page spreads found byequation (8) is satisfied. Due to this, it is possible to set the numberof common double-page spreads that is in accordance with both the numberof individual double-page spreads and the total number of double-pagespreads.

Here, the case is explained where the number of double-page spreads isset in accordance with the number of candidate images, but the settingmethod of the number of double-page spreads is not limited to this, Forexample, it may also be possible for the double-page spread creationparameter generation unit 209 to set the number of double-page spreadswithin the parameter and for the number of double-page spreads settingunit 21003 to read the number of double-page spreads set within theparameter. Further, for example, it may also be possible for thedouble-page spread creation parameter generation unit 209 to set thenumber of double-page spreads incorporated in advance in the program ofthe album creation application or calculate the number of double-pagespreads based on the ratio between the individual double-page spreadspecified by a user and the common double-page spread.

At S51106, a double-page spread allocation unit 21004 divides thecandidate images acquired at S51102 into sub image groups correspondingto the number of double-page spreads set at S51104 or S51105 andallocates the sub image groups. In the present embodiment, the candidateimages are arranged in order of the image capturing time acquired atS50301 and the candidate images are divided at the position where thetime difference in the image capturing time between adjacent images islarge. The processing such as this is performed until the candidateimages are divided into sub image groups corresponding to the number ofdouble-page spreads set at S51104 or S51105. That is, the division isperformed (the number of double-page spreads−1) times. Due to this, itis possible to create an album in which the images are arranged in orderof the image capturing time.

A S51107, a double-page spread image selection unit 21005 selects imagesthat are used for the layout from among the candidate image groupallocated to the processing-target double-page spread at S51106. In thefollowing, by using FIG. 8, explanation is given by taking a case as anexample where four images are selected from among the candidate imagegroup allocated to a certain double-page spread. Here, it is assumedthat the double-page spread refers to two pages and the firstdouble-page spread and the last double-page spread include one page. Thenumber of images selected as images used for the layout is not limitedto four. The number of images selected as images used for the layout isdetermined, for example, in accordance with the number of images that isset in the number of double-page spread photos setting unit 406.Specifically, the number of images may be determined in such a mannerthat in a case where “small” at the first level is set by the slider barof the number of double-page spread photos setting unit 406, the numberof images is determined to be one, in a case of the second level, it isdetermined to be two, in a case of the third level, it is determined tobe three, in a case of the fourth level, it is determined to be four, ina case where “large” at the fifth level is set, it is determined to befive, and so on.

In FIG. 8, (a) indicates the time difference in the image capturing datefrom the first image included in the candidate image group allocated tothe double-page spread to the last image, in other words, the imagecapturing period of the candidate image group. By using (b) in FIG. 8, amethod of selecting the first image at the time of selecting four imagesis explained. From among all the images captured during the imagecapturing period of the candidate image group shown in (b) in FIG. 8,the image whose score for which re-scoring has been performed at S51102is the highest is selected as the first image. In selecting the secondand subsequent images, by selecting images by dividing the imagecapturing period of the candidate image group into smaller periods, theselected images are prevented from concentrating on a portion of theimage capturing period of the candidate image group. First, as shown in(c), the image capturing period of the candidate image group is halvedinto two image capturing sections (periods). That is, the imagecapturing period is halved (grouped) into two image capturing sections.Next, as shown in (d) in FIG. 8, the image whose score is the highestamong the images belonging to the image capturing section (sectionindicated by a solid line) in which the first image is not selected isselected as the second image. Next, as shown in (e) in FIG. 8, eachimage capturing section in (d) is halved. The image whose score is thehighest is selected as the third image from among the images in theimage capturing sections indicated by a solid line in (f) in FIG. 8,that is, the images captured in the two image capturing sections inwhich neither first image nor second image is selected (imagescorresponding to the two image capturing sections). Next, a case whereno image exists in the image capturing section of the range in which animage is to be selected and an image cannot be selected is explained bytaking selection of the fourth image as an example. In the presentembodiment, regardless of the number of images, the image capturingperiod is divided in accordance with time. Consequently, there is a casewhere no image exists in the image capturing section obtained bydivision. For example, it is assumed that although it is desired toselect the fourth image from the image capturing section (imagecapturing section indicated by slashes) in which no image is selectedyet, no image exists in the image capturing section as shown in (g) inFIG. 8. In such a case, as shown in (h) in FIG. 8, each image capturingsection in which an image has already been selected is halved. Next, asshown in (i) in FIG. 8, the image whose score is the highest is selectedas the fourth image from among the images captured in the imagecapturing sections indicated by a solid line, in which none of the firstto third images is selected. In a case where the number of images to beselected increases or decreases from four, images are selected similarlyas described previously.

Explanation is returned to FIG. 5C. At S51108, a template setting unit21006 selectively sets a plurality of templates to be used for thelayout from among the templates saved in the HDD 104. At this step,templates with a design satisfying the album creation conditions set atS501 are set.

At S51109, a double-page spread layout unit 21007 determines an imagelayout for the processing-target double-page spread. Specifically, thedouble-page spread layout unit 21007 determines a template suitable forlaying out the images selected at S51107 from among the plurality oftemplates set at S51108. Here, a method of determining a template atthis step is explained by using FIG. 9. FIG. 9 is a function blockdiagram relating to template determination.

A selected image information acquisition unit 901 acquires the number ofimages selected by the double-page spread image selection unit 21005 andinformation relating to the images. The image information that isacquired is the width and height of the image, the image capturing dateinformation, and the score calculated by the image scoring unit 208. Atemplate selection unit 902 selects templates having the number of slotsequal to the number of selected images from among the templates set bythe template setting unit 21006. A selected image sort unit 903 sortsthe selected images in order of the image capturing time. A templateselection unit 904 selects templates whose order number of the main slotin which the main image is arranged matches with the order number of themain image in a case where the images are arranged in order of the imagecapturing time from among the templates selected by the templateselection unit 902. Here, the “main image” is the image whose scoreobtained as a result of re-scoring at S51102 is the highest among theplurality of images selected as the double-page spread images and on theother hand, the image that is not the main image among the plurality ofimages is referred to as “sub image”. At the time of selecting atemplate, in addition to the order number, that the aspect ratio of themain image and the aspect ratio of the slot in which the main image isarranged match with each other is the selection condition. In thepresent embodiment, it is assumed that at the top left of the template,the image whose image capturing time is older is laid out and at thebottom right, the image whose image capturing time is newer is laid out.A template selection unit 905 selects a template from among thetemplates selected by the template selection unit 904, whose ordernumber of the slot in which the sub image is arranged matches with theorder number of the sub image in a case where the images are arranged inorder of the image capturing time and whose aspect ratio of the slotmatches with the aspect ratio of the image. FIG. 10 is a flowchart oftemplate determination processing.

At S1001, the selected image information acquisition unit 901 acquiresthe selected image information. In the selected image information thatis acquired at this step, the number of selected images is included.Here, for explanation, it is assumed that the number of selected imagesis three.

At S1002, the template selection unit 902 selects templates whose numberof slots matches with the number of selected images. Here, it is assumedthat the number of selected images acquired at S1001 is three and thetemplates whose number of slots is three are selected. In the following,explanation is given by taking a case where templates of (1-1) to (4-4)shown in FIG. 11 are selected at this step as an example.

At S1003, the selected image sort unit 903 arranges the selected imagesin order of the image capturing time. Here, it is assumed that arelationship shown in (A) in FIG. 11 is obtained as a result ofarranging the selected images in order of the image capturing time.Further, it is assumed that an image 1105 is the main image and an image1106 and an image 1107 are the sub images.

At S1004, the template selection unit 904 selects templates whosearrangement position of the slot for the main image (referred to as mainslot) matches with the arrangement position of the main image and whoseaspect ratio of the main slot matches with the aspect ratio of the mainimage. Here, the image 1105 is the image whose image capturing time isthe newest and which is the image for the main slot, and therefore, thetemplate candidates are (3-1) to (3-4) in FIG. 11.

At S1005, the template selection unit 905 selects a template whoseaspect ratio of the slot for the sub image (referred to as sub slot)matches with that of the sub image. In the example in FIG. 11, the olderimage 1106 for the sub slot, that is, the image desired to be arrangedat the top left is the portrait image and the newer image 1107 for thesub slot is the landscape image. Consequently, the template of (3-2) isdetermined as the template the most suitable for the three selectedimages. As described above, at S51109, the template used for the layoutand the information capable of identifying which image is laid out inwhich slot of the template are determined.

Explanation is returned to FIG. 5C. At S51110, the double-page spreadlayout unit 21007 determines whether the processing at S51107 to S51109has been completed for all the processing-target double-page spreads. Ina case where determination results at this step are affirmative, theprocessing advances to S51111 and on the other hand, in a case where thedetermination results are negative, the processing returns to S51107.

At S51111, the double-page spread creation parameter setting unit 21001determines whether the processing at S51101 to S51110 has been completedfor all the parameters generated at S510. In a case where determinationresults at this step are affirmative, the processing advances to S512and on the other hand, in a case where the determination results arenegative, the processing returns to S51101.

Explanation is returned to FIG. 5A. At S512, the double-page spreadcombination unit 211 determines the double-page spread number within thealbum for each double-page spread created by the double-page spreadcreation unit 210 and creates the layout of the entire album. In thepresent embodiment, the double-page spreads are arranged so that theimages are arranged in order of the image capturing time based on theimage capturing times (image capturing dates) of the images arranged onthe double-page spreads.

In the following, double-page spread combination at S512 is explained indetail by using FIG. 12 and FIG. 13. FIG. 12 is a block diagram showinga detailed configuration of the double-page spread combination unit 211in the present embodiment. FIG. 13 is a flowchart of processingperformed by the double-page spread combination unit 211.

At S1301, a double-page spread acquisition unit 1201 acquires thedouble-page spreads relating to one generation-target album among thedouble-page spreads created by the double-page spread creation unit 210.In the present embodiment, for example, the double-page spreads relatingto the album 1, which are acquired at this step, include a double-pagespread created with the common parameter and a double-page spreadcreated with the individual parameter of the album 1.

At S1302, a sort reference value calculation unit 1202 calculates avalue that is a reference of sort for each double-page spread acquiredby the double-page spread acquisition unit 1201. In the presentembodiment, it is assumed that the reference value of sort is theaverage image capturing time (average of image capturing date of eachimage) of the images arranged on the double-page spread. The referencevalue of sort is not limited to the image capturing time. For example,the reference value of sort may be the average image capturing position(latitude and longitude) of the images arranged on the double-pagespread or the number of images, or it may also be possible to use imageanalysis information, such as a color histogram.

At S1303, a sort unit 1203 sorts the double-page spreads acquired by thedouble-page spread acquisition unit 1201 based on the sort referencevalue calculated by the sort reference value calculation unit 1202. Thatis, the sort unit 1203 determines the arrangement order of thedouble-page spreads acquired by the double-page spread acquisition unit1201. In the present embodiment, the double-page spreads are arranged inascending order from the double-page spread whose average imagecapturing time is the earliest.

At S1304, a number of double-page spreads adjustment unit 1204determines whether the number of double-page spreads of the album is thesame as the total number of double-page spreads set at S501. In a casewhere determination results at this step are affirmative, the processingadvances to S1306. On the other hand, in a case where determinationresults at this step are negative, that is, in a case where the numberof double-page spreads of the album exceeds the total number ofdouble-page spreads as a result of the number of double-page spreads ofeach album being set to a number larger than or equal to the totalnumber of double-page spreads at S51103, the processing advances toS1305.

At S1305, the number of double-page spreads adjustment unit 1204performs adjustment so that the number of double-page spreads of thealbum becomes close to the total number of double-page spreads set atS501 by deleting the unnecessary double-page spread of the double-pagespreads acquired by the double-page spread acquisition unit 1201. In thepresent embodiment, the number of double-page spreads is adjusted bydeleting the common double-page spread. Specifically, among the commondouble-page spreads located within the album, the common double-pagespread whose image capturing time of the double-page spread calculatedby the sort reference value calculation unit 1202 is the closest to theimage capturing time of the individual double-page spread is deleted.

After the deletion, the processing advances to S1304. In the presentembodiment, the common double-page spread is deleted, but this is notlimited. It may also be possible to adjust the total number ofdouble-page spreads by deleting the individual double-page spread. Thatis, it is sufficient to delete part of the common double-page spreads orthe individual double-page spreads.

At S1306, the double-page spread acquisition unit 1201 determineswhether the processing at S1301 to S1305 has been completed for all thealbums. In a case where determination results at this step areaffirmative, the processing advances to S513 and on the other hand, in acase where the determination results are negative, the processingreturns to S1301. Due to this, it is possible to arrange the images inorder of the image capturing time throughout the entire album. The aboveis the contents of the automatic layout processing in the presentembodiment. In the present embodiment, the processing is performed inunits of spread pages, but the processing may be performed in units ofpages. That is, it may also be possible to create the “individual page”and the “common page” in units of double-page spreads (that is, twopages) or create the “individual page” and the “common page” in units ofpages.

<Effects of the Present Embodiment>

FIG. 29 is a diagram for explaining effects of the present embodimentand shows examples of a plurality of variations of album created in thepresent embodiment. To explain in detail, FIG. 29 shows a case where thealbums 1 to 3 are created as album data for the objects A to C. The mainobject in the album 1 is set to the object A, the main object in thealbum 2 is set to the object B, and the main object in the album 3 isset to the object C, respectively. A common double-page spread 2901represents a common double-page spread that is not used finally in thealbum 2 as a result of adjustment (NO at S1304→S1305 in FIG. 13) becausethe acquired number of double-page spreads exceeds the total number ofdouble-page spreads.

As shown in FIG. 29, as regards the variations of album created in thepresent embodiment, the total number of double-page spreads does notchange depending on the album and is constant at all times. The reasonis to eliminate a sense of unfairness resulting from the total number ofdouble-page spreads differing for each album. In the example in FIG. 29,the total number of double-page spreads is four irrespective of thealbum.

Further, in the present embodiment, after the score of image quality andthe score for each object are given for one image, the image is furtherre-scored based on the individual parameter. Consequently, as a resultof re-scoring, the number of candidate images of the individualdouble-page spread changes depending on the album, and therefore, thenumber of individual double-page spreads may change. However, in thepresent embodiment, although the number of good photos varies for eachobject, priority is given to improvement of the degree of satisfactionof a user by using good photos for each object as much as possible.Consequently, it is permitted for the number of individual double-pagespreads to change depending on the album. In the example in FIG. 29, thenumber of individual double-page spreads of the album 1 and the album 3is one, but on the other hand, the number of individual double-pagespreads of the album 2 is two.

Further, in the present embodiment, at S1303, the double-page spreadsare sorted based on the predetermined sort reference value. For example,the double-page spreads are sorted in a time series. That is, by thissort, arrangement order of the double-page spreads is determined.Consequently, in the present embodiment, the position of the individualdouble-page spread for each piece of album data may change. That is, itis possible to generate album data in which the position of theindividual page is different for each piece of album data. However, inthe present embodiment, priority is given to that the degree ofsatisfaction of a user is improved as an album in a case where thedouble-page spreads are arranged in accordance with a predetermined sortreference, such as a time series, and therefore, it is permitted alsofor the position of the individual double-page spread of each album tochange. In the example in FIG. 29, the results of a case are shown wherethe position of the individual double-page spread is the same by chancein all the albums. For example, in a case where the average imagecapturing date of the individual double-page spread of the object A ofthe album 1 is earlier than that of any of the common double-pagespreads, the individual double-page spread of the object A is arrangedon the first page of the album.

Further, in the present embodiment, the total number of double-pagespreads is set so as not to change for each album, but the number ofindividual double-page spreads may change depending on the album. Thatis, the number of common double-page spreads may also change dependingon the album. However, priority is given to that the degree ofsatisfaction of a user is improved by using good photos as much aspossible as described above, and therefore, it is permitted for thenumber of common double-page spreads to change depending on the album.In the example in FIG. 29, the number of common double-page spreads ofthe album 1 and the album 3 is three, but on the other hand, the numberof common double-page spreads of the album 2 is two.

According to the present embodiment, it is made possible toautomatically set the number of common double-page spreads, the numberof individual double-page spreads, and the positions thereof for eachalbum.

Second Embodiment

In the present embodiment, a double-page spread common to at least twoalbums and an individual double-page spread in each album are created.The total number of double-page spreads, the number of individualdouble-page spreads, and the positions of the individual double-pagespreads for each album change in accordance with the image group and thesetting. The images arranged within the double-page spread are arrangedin order of the image capturing time. In the following explanation,explanation of the portions in common to those of the first embodimentis omitted and portions different from the first embodiment areexplained mainly.

<Automatic Layout Processing>

In the following, the automatic layout processing in the presentembodiment is explained by using FIG. 14A, FIG. 14B, and FIG. 15A toFIG. 15C. FIG. 14A and FIG. 14B are each a block diagram showing thefunction configuration relating to the automatic layout processing inthe present embodiment. FIG. 15A to FIG. 15C are each a flowchart forexplaining the automatic layout processing in the present embodiment.The processing that is indicated by the same symbol as that of theembodiment described previously (see FIG. 5A to FIG. 5C) is the sameprocessing as that of the embodiment described previously, andtherefore, explanation is omitted here.

At S1501, a double-page spread creation unit 1401 creates a double-pagespread of the album by using the parameters generated at S510. Here,double-page spread creation at this step is explained by using FIG. 15C.

At S150101, a number of double-page spreads setting unit 140101 sets thenumber of double-page spreads based on the results of the scene divisionperformed at S505 and the candidate images acquired at S51102. In thepresent embodiment, the number of double-page spreads is set inaccordance with equation (9) and equation (10) below.number of double-page spreads for each scene=ceil(number of candidateimages in scene÷maximum number of images per double-page spread)  (9)number of double-page spreads=Σnumber of double-page spreads for eachscene  (10)

Here, ceil( ) s the function that returns the minimum integer valuelarger than or equal to the argument. By equation (9) and equation (10),it is possible to set the number of double-page spreads in accordancewith the number of images in which the main object is capturedfrequently for each scene.

At S1502, a double-page spread combination unit 1402 determines thedouble-page spread number within the album for each double-page spreadcreated by the double-page spread creation unit 1401 and creates thelayout of the entire album. In the present embodiment, based on theimage capturing times of the images arranged on the double-page spread,the double-page spreads are arranged so that the images are arranged inorder of the image capturing time. In the following, double-page spreadcombination at S1502 is explained in detail by using FIG. 16 and FIG.17. FIG. 16 is a block diagram showing a detailed configuration of thedouble-page spread combination unit 1402 in the present embodiment. FIG.17 is a flowchart of processing performed by the double-page spreadcombination unit 1402.

At S1301, the double-page spread acquisition unit 1201 acquiresdouble-page spreads relating to one album among the double-page spreadscreated by the double-page spread creation unit 1401. In the presentembodiment, the double-page spreads relating to the album 1, which areacquired at this step, include, for example, the double-page spreadcreated with the common parameter and the double-page spread createdwith the individual parameter of the album 1.

At S1302, the sort reference value calculation unit 1202 calculates theimage capturing time representing each double-page spread for eachdouble-page spread acquired by the double-page spread acquisition unit1201. In the present embodiment, it is assumed that the image capturingtime representing the double-page spread is the average of the imagecapturing times of the images arranged on the double-page spread.

At S1303, the sort unit 1203 sorts the double-page spreads acquired bythe double-page spread acquisition unit 1201 based on the imagecapturing time of the double-page spread, which is calculated by thesort reference value calculation unit 1202. In the present embodiment,the double-page spreads are arranged in ascending order from thedouble-page spread whose image capturing time is the earliest.

At S1701, the double-page spread acquisition unit 1201 determineswhether the processing at S1301 to S1303 has been completed for all thealbums. In a case where determination results at this step areaffirmative, the processing advances to S513 and on the other hand, in acase where the determination results are negative, the processingreturns to S1301. Due to this, it is possible to arrange the images inorder of the image capturing time throughout the entire album. The aboveis the contents of the automatic layout processing in the presentembodiment.

In the present embodiment, the processing is performed in units ofspread pages, but the processing may be performed in units of pages.That is, it may also be possible to create the “individual page” and the“common page” in units of double-page spreads (that is, two pages) orcreate the “individual page” and the “common page” in units of pages.

<Effects of the Present Embodiment>

FIG. 30A and FIG. 30B are diagrams for explaining effects of the presentembodiment and show examples of a plurality of variations of albumcreated in the present embodiment. To explain in detail, FIG. 30A showsa case where the albums 1 and 2 are created as albums for the objects Aand B and the main object in the album 1 is set to the object A and themain object in the album 2 is set to the object B. Further, FIG. 30Bshows a case where the albums 1 to 3 are created as albums for theobjects A to C and the main object in the album 1 is set to the objectA, the main object in the album 2 is set to the object B, and the mainobject in the album 3 is set to the object C, respectively.

In the present embodiment, different from the first embodiment, therestriction that the total number of double-page spreads be the same inthe plurality of albums is removed. The reason is that in the presentembodiment, the number of photos is not limited and priority is given toimprovement of the degree of satisfaction of a ser by using good photosas much as possible. Consequently, as regards the variations of album tobe created, it is permitted for the total number of double-page spreadsto change depending on the album. In the example in FIG. 30A, the totalnumber of double-page spreads is four by chance irrespective of thealbum and in the example in FIG. 30B, the total number of double-pagespreads is three irrespective of the album. That is, the examples areshown in which the total number of double-page spreads is the same inthe plurality of albums, but for example, in FIG. 30A, in a case wherethe number of individual double-page spreads of the object A of thealbum 1 is two, the total number of double-page spreads is differentbetween the album 1 and the album 2.

Further, in the present embodiment, as described above, priority isgiven to improvement of the degree of satisfaction of a user by usinggood photos as much as possible. Consequently, it is permitted for thenumber of individual double-page spreads to change depending on thealbum. In the example in FIG. 30A, the number of individual double-pagespreads of the album 1 and the album 2 is one and in the example in FIG.30B, the number of individual double-page spreads of the albums 1 to 3is one, and both the examples show the results in which the number ofindividual double-page spreads is the same by chance in all the albums.However, for example, in a case where the number of individualdouble-page spreads of the object A of the album 1 is two in FIG. 30A,the number of individual double-page spreads is different between thealbum 1 and the album 2.

Further, in the present embodiment, at S1303, the double-page spreadsare sorted based on the predetermined sort reference value. For example,the double-page spreads are sorted in a time series. That is, by thissort, the arrangement order of the double-page spreads is determined.Consequently, in the present embodiment, the position of the individualdouble-page spread of each album may change. However, in the presentembodiment, priority is given to that the degree of satisfaction of auser is improved as an album in a case where the double-page spreads arearranged in accordance with a predetermined sort reference, such as atime series, and therefore, it is also permitted for the position of theindividual double-page spread of each album to change. The examples inFIG. 30A and FIG. 30B show the results of a case where the position ofthe individual double-page spread is the same by chance. For example, ina case where the average image capturing date of the individualdouble-page spread of the object A of the album 1 is earlier than thatof any of the common double-page spreads, the individual double-pagespread of the object A is arranged on the first page of the album.

Further, in the present embodiment, different from the first embodiment,the restriction that the total number of double-page spreads be the samein the plurality of albums is removed. Because of this, there is no stepof deleting the common double-page spread in a case where the totalnumber of double-page spreads is exceeded. Consequently, in the presentembodiment, the number of common double-page spreads does not changedepending on the album and is the same at all times. As shown in FIG.30A, the number of common double-page spreads of the album 1 and thealbum 2 is three. Further, as shown in FIG. 30B, the number of commondouble-page spreads of the albums 1 to 3 is two.

According to the present embodiment, it is made possible toautomatically set the total number of double-page spreads, the number ofindividual double-page spreads, and the positions thereof of the albumfor each album.

Third Embodiment

In the present embodiment, a double-page spread common to at least twoalbums and an individual double-page spread in each album are created.The number of common double-page spreads and the number of individualdouble-page spreads are made the same for all the albums. However, theposition of the individual double-page spread changes for each album inaccordance with the image group and the setting. In each album, thedouble-page spreads and the images that are arranged within thedouble-page spread are arranged in order of the image capturing time. Inthe following explanation, explanation of the portions in common tothose of the first embodiment and the second embodiment is omitted andportions different from the first embodiment and the second embodimentare explained mainly.

<Automatic Layout Processing>

In the following, the automatic layout processing in the presentembodiment is explained by using FIG. 18A, FIG. 18B, and FIG. 19A toFIG. 19C. FIG. 18A and FIG. 18B are each a block diagram showing thefunction configuration relating to the automatic layout processing inthe present embodiment. FIG. 19A to FIG. 19C are each a flowchart forexplaining the automatic layout processing in the present embodiment.The processing that is indicated by the same symbol as that of theembodiment described previously is the same processing as that of theembodiment described previously, and therefore, explanation is omittedhere.

At S1901, a double-page spread creation unit 1801 creates thedouble-page spread of the album by using the parameters generated atS510. Here, double-page spread creation processing at this step isexplained by using FIG. 19C.

At S190101, a number of double-page spreads calculation unit 180101calculates the number of double-page spreads created with each parameterbased on the parameters generated at S510 and sets the number ofdouble-page spreads to each double-page spread creation parameter. Itmay also be possible to calculate and set the number of pages createdwith each parameter. In the following, processing performed by thenumber of double-page spreads calculation unit 180101 in the presentembodiment is explained by using FIG. 20.

At S2001, the double-page spread creation parameter setting unit 21001reads one parameter that makes the same the number of double-pagespreads among the parameters generated at S510. In the presentembodiment, the individual parameter of the album 1 and the individualparameter of the album 2 are read in order.

At S2002, the double-page spread candidate image acquisition unit 21002acquires candidate images by the same method as that at S51102 based onthe score given at S508 and the double-page spread creation parameterset at S2001.

At S2003, the number of double-page spreads calculation unit 180101calculates the candidate number of individual double-page spreads basedon the candidate images acquired at S2002. In a case where the number ofpages is set, the candidate number of individual pages is calculated. Inthe present embodiment, the candidate number of individual double-pagespreads is calculated in accordance with equation (11) below.candidate number of individual double-page spreads=ceil(number ofcandidate images÷maximum number of images per double-page spread)  (11)

The candidate number of individual double-page spreads calculated byusing equation (11) increases or decreases according to the number ofcandidate images.

At S2004, the number of double-page spreads calculation unit 180101determines whether the processing at S2001 to S2003 has been completedfor all the target parameters that make the same the number ofdouble-page spreads. In a case where determine results at this step areaffirmative, the processing advances to S2005 and on the other hand, ina case where the determine results are negative, the processing returnsto S2001.

At S2005, the number of double-page spreads calculation unit 180101determines the number of double-page spreads based on a candidate numberof individual double-page spreads group calculated at S2003. In thepresent embodiment, the smallest number of double-page spreads isselected from among the candidate numbers of individual double-pagespreads. Due to this, it is possible to create the same number ofindividual double-page spreads in accordance with the parameter with asmall number of candidate images. Further, the number of commondouble-page spreads created with the common parameter is determinedsubordinately based on the number of individual double-page spreadscreated with the individual parameter and the total number ofdouble-page spreads set at S501 as indicated by equation (12) below.number of common double-page spreads=total number of double-pagespreads−number of individual double-page spreads  (12)

At S2006, the number of double-page spreads calculation unit 180101updates the double-page spread creation parameter by adding the numberof double-page spreads determined at S2005 to the parameters generatedat S510.

As described above, at S190101, the number of individual double-pagespreads and the number of common double-page spreads are adjusted, andtherefore, S1304 and S1305 at S512 may be omitted.

Explanation is returned to FIG. 19C. At S190102, a number of double-pagespreads setting unit 180102 reads the number of double-page spreads setat S190101 from the double-page spread creation parameter read atS51101.

At S190103, a double-page spread allocation unit 180103 allocates thecandidate images acquired at S51102 to the sub image groupscorresponding to the number of double-page spreads set at S190102. Then,in the present embodiment, further, from the sub image groups to whichthe candidate images are allocated, the sub image groups correspondingto the number of double-page spreads included in the double-page spreadcreation parameter set at the immediately previous S51101 are selectedin order from the sub image group whose average of the image score isthe highest.

In the present embodiment, the number of double-page spreads is set inaccordance with the number of candidate images, but the method ofsetting the number of double-page spreads is not limited to this. Forexample, it may also be possible to set the number of individualdouble-page spreads and the number of common double-page spreads by aratio incorporated in advance in the program of the album creationapplication. Specifically, in a case where the number of individualdouble-page spreads: the number of common double-page spreads=1:4 isset, on a condition that the total number of double-page spreads set atS501 is ten, the number of individual double-page spreads is set to twoand the number of common double-page spreads is set to eight. Further,as another example, it may also be possible to determine the number ofdouble-page spreads by the scene classification results at S506.

For example, a case is considered where the scene classification resultsis the wedding. It may also be possible to set a plurality of thenumbers of individual double-page spreads in accordance with the scene,such as a total of ten double-page spreads, that is, two double-pagespreads with the individual parameter setting the groom and bride as themain objects, one double-page spread with the individual parametersetting relatives or friends as the main objects, and seven double-pagespreads with the common parameter. In the example described above, itmay also be possible for the number of double-page spreads calculationunit 180101 to increase or decrease the threshold value for acquiringcandidate images so as to match with the number of double-page spreads,in addition to the number of double-page spreads. For example, a case isconsidered where it is desired to create two or more double-page spreadsalthough only one candidate image can be acquired with the double-pagespread creation parameter acquired at S2001. Until it is made possiblefor the double-page spread candidate image acquisition unit 21002 toacquire two or more candidate images, the threshold value of the score,which is the selection reference, is reduced. The double-page spreadcreation parameter is updated by adding the changed threshold value tothe parameters generated at S510 as in the case with the number ofdouble-page spreads. Due to this, it is possible to acquire candidateimages in accordance with the number of double-page spreads requested bya user. The above is the contents of the automatic layout processing inthe present embodiment. In the present embodiment, the processing isperformed in units of spread pages, but the processing may be performedin units of pages. That is, it may also be possible to create the“individual page” and the “common page” in units of double-page spreads(that is, two pages) or create the “individual page” and the “commonpage” in units of pages.

<Effects of the Present Embodiment>

FIG. 31A and FIG. 31B are diagrams for explaining effects of the presentembodiment and show examples of a plurality of variations of albumcreated in the present embodiment. To explain in detail, FIG. 31A showsa case where the albums 1 and 2 are created as albums for the objects Aand B and the main object in the album 1 is set to the object A and themain object in the album 2 is set to the object B. Further, FIG. 31Bshows a case where the albums 1 to 3 are created as albums for theobjects A to C and the main object in the album 1 is set to the objectA, the main object in the album 2 is set to the object B, and the mainobject in the album 3 is set to the object C, respectively.

In the present embodiment, as regards the variations of album to becreated, the total number of double-page spreads does not changedepending on the album and is the same at all times. The reason is toeliminate a sense of unfairness resulting from the total number ofdouble-page spreads varying for each album. In the example in FIG. 31A,the total number of double-page spreads is three irrespective of thealbum and in the example in FIG. 31B, the total number of double-pagespreads is two irrespective of the album.

Further, in the present embodiment, for example, the album is created inaccordance with the smallest number of double-page spreads among thecandidate numbers of individual double-page spreads. Consequently, thenumber of individual double-page spreads does not change depending onthe album and is the same at all times. Due to this, it is possible toeliminate a sense of unfairness resulting from the number of individualdouble-page spreads varying for each album. In the example in FIG. 31A,the number of individual double-page spreads of the album 1 and thealbum 2 is two and in the example in FIG. 31B, the number of individualdouble-page spreads of the albums 1 to 3 is one.

Further, in the present embodiment, in each album, the individualdouble-page spreads are arranged in chronological order. Consequently,in the present embodiment, for each album, the position of theindividual double-page spread may change. However, in the presentembodiment, priority is given to that the degree of satisfaction of auser is improved as an album in a case where the double-page spreads arearranged in a time series, and therefore, it is also permitted for theposition of the individual double-page spread of each album to change.The examples in FIG. 31A and FIG. 31B show the results of a case wherethe position of the individual double-page spread is the same by chancein all the albums.

Further, in the present embodiment, the number of common double-pagespreads is found based on, for example, the total number of double-pagespreads and the number of individual double-page spreads. Consequently,the number of common double-page spreads does not change depending onthe album and is the same at all times. In the example in FIG. 31A, thenumber of common double-page spreads of the album 1 and the album 2 isone and in the example in FIG. 31B, the number of common double-pagespreads of the albums 1 to 3 is one.

According to the present embodiment, it is made possible to make thesame the total number of double-page spreads, the number of commondouble-page spreads, and the number of individual double-page spreads inall the albums.

Fourth Embodiment

In the present embodiment, a double-page spread common to at least twoalbums and an individual double-page spread in each album are created.The number of common double-page spreads, the positions of the commondouble-page spreads, the number of individual double-page spreads, andthe positions of the individual double-page spreads are made the same inall the albums. Within each album, the common double-page spreads andthe images that are arranged within the common double-page spread arearranged in order of the image capturing time. The position of theindividual double-page spread is determined by the image irrespective ofthe image capturing time. In the following explanation, explanation ofthe portions in common to those of the first to third embodiments isomitted and portions different from the first to third embodiment areexplained mainly.

<Automatic Layout Processing>

In the following, the automatic layout processing in the presentembodiment is explained by using FIG. 21A, FIG. 21B, and FIG. 22A toFIG. 22C. FIG. 21A and FIG. 21B are each a block diagram showing thefunction configuration relating to the automatic layout processing inthe present embodiment. FIG. 22A to FIG. 22C are each a flowchart forexplaining the automatic layout processing in the present embodiment.The processing that is indicated by the same symbol as that of theembodiment described previously is the same processing as that of theembodiment described previously, and therefore, explanation is omittedhere.

At S2201, a double-page spread creation unit 2101 creates a double-pagespread of the album by using the parameters generated at S510. Here,double-page spread creation processing at this step is explained byusing FIG. 22C.

At S220101, a number of double-page spreads setting unit 210101 readsthe number of double-page spreads set at S190101 from the double-pagespread creation parameter read at S51101.

At S220102, a double-page spread allocation unit 210102 divides thecandidate images acquired at S51102 into sub image groups correspondingto the number of double-page spreads set at S220101 and allocates thesub image groups. In the present embodiment, in a case where thedouble-page spread creation parameter set at the immediately previousS51101 is the common parameter, the candidate images are divided basedon the image capturing time acquired at S50301. On the other hand, in acase where the double-page spread creation parameter set at theimmediately previous S51101 is the individual parameter, the candidateimages are divided based on the score given at S508.

Specifically, in a case where the double-page spread creation parameterset at the immediately previous S51101 is the common parameter, thecandidate images are arranged in order of the image capturing timeacquired at S50301 and the candidate images are divided at the positionwhere the time difference between adjacent images is the largest. Thedivision processing such as this is performed until the candidate imagesare divided into sub image groups corresponding to the number ofdouble-page spreads set at S220101. That is, the division is performed(the number of double-page spreads−1) times. Due to this, it is possibleto create an album in which the images are arranged in order of theimage capturing time. As will be described later, within one double-pagespread, the images do not need to be arranged in order of the imagecapturing time.

Further, in a case where the double-page spread creation parameter setat the immediately previous S51101 is the individual parameter, thecandidate images are allocated to each double-page spread in order fromthe candidate image whose score is the highest. For example, in a casewhere the number of double-page spreads is three, the candidate imagesare allocated to double-page spread 1, double-page spread 2, double-pagespread 3, double-page spread 1, double-page spread 2, . . . , and so on,in order from the candidate image whose score is the highest. Due tothis, as a result of the images whose score is high being equallyallocated to each double-page spread, it is possible to arrange theimages suitable for the main objects in a balanced manner, not in a timeseries.

Explanation is returned to FIG. 22A. At S2202, a double-page spreadcombination unit 2102 determines the double-page spread number withinthe album for each double-page spread created by the double-page spreadcreation unit 2101 and creates the layout of the entire album. In thepresent embodiment, the common double-page spreads created based on theimage capturing time are arranged in order of the image capturing time.On the other hand, the individual double-page spread created based onthe score is inserted at the position where the time difference betweencommon double-page spreads is large.

In the following, double-page spread combination at S2202 is explainedby using FIG. 23 and FIG. 24. FIG. 23 is a block diagram showing adetailed configuration of the double-page spread combination unit 2102.FIG. 24 is a flowchart of processing performed by the double-page spreadcombination unit 2102.

At S2401, a double-page spread acquisition unit 2301 acquiresdouble-page spreads relating to one album from among the double-pagespreads created by the double-page spread creation unit 2101. In thepresent embodiment, for example, the double-page spreads relating to thealbum 1, which are acquired at this step, include the common double-pagespread created with the common parameter and the individual double-pagespread created with the individual parameter of the album 1.

At S2402, a sort reference value calculation unit 2302 calculates animage capturing time representing each double-page spread for eachdouble-page spread acquired by the double-page spread acquisition unit2301. In the present embodiment, the average, the minimum, and themaximum of the image capturing times of the images arranged on thedouble-page spread are taken as three image capturing times representingthe double-page spread.

At S2403, a sort unit 2303 sorts the double-page spreads acquired by thedouble-page spread acquisition unit 2301 based on the image capturingtime of the double-page spread, which is found by the sort referencevalue calculation unit 2302. In the present embodiment, the commondouble-page spreads are arranged in ascending order from the commondouble-page spread whose average of the image capturing time is theoldest.

At S2404, an individual double-page spread insertion unit 2304determines the double-page spread number of the individual double-pagespread among the double-page spreads acquired by the double-page spreadacquisition unit 2301. In the present embodiment, for the commondouble-page spreads sorted at S2403, the individual double-page spreadis inserted at the position where the image capturing time differencebetween common double-page spreads is the maximum. The image capturingtime difference between the Nth common double-page spread and the(N+1)th common double-page spread is calculated in accordance withequation (13) below.minimum image capturing time of(N+1)th common double-page spread−maximumimage capturing time of Nth common double-page spread  (13)

By equation (13), it is possible to insert the individual double-pagespread at the position where the image capturing time difference islarge (that is, the possibility that the scene has changed is strong).Here, the case is explained where all the individual double-page spreadsare inserted between the common double-page spreads where the imagecapturing time difference is the maximum, but the individual double-pagespread insertion method is not limited to this. For example, it may alsobe possible to insert the individual double-page spreads at differentpositions, such as that the ith individual double-page spread isinserted between common double-page spreads where the image capturingtime difference is the ith largest. Further, it may also be possible toinsert the individual double-page spreads at the position where thecommon double-page spreads are not divided irrespective of the timedifference, such as the position before all the common double-pagespreads and the position after all the common double-page spreads.Furthermore, in a case where all the time differences between commondouble-page spreads are smaller than the predetermined value, it mayalso be possible to insert the individual double-page spreads at aposition where the common double-page spreads are not divided, such asthe position before all the common double-page spreads and the positionafter all the common double-page spreads.

At 2405, the double-page spread acquisition unit 2301 determines whetherthe processing at S2401 to S2404 has been completed for all the albums.In a case where determination results at this step are affirmative, theprocessing advances to S513 and on the other hand, in a case where thedetermination results are negative, the processing returns to S2401. Dueto this, it is possible to arrange the common double-page spreads andthe individual double-page spreads at the positions whose double-pagespread number is the same in all the albums. The above is the contentsof the automatic layout processing in the present embodiment. In thepresent embodiment, the processing is performed in units of spreadpages, but the processing may be performed in units of pages. That is,it may also be possible to create the “individual page” and the “commonpage” in units of double-page spreads (that is, two pages) or create the“individual page” and the “common page” in units of pages.

<Effects of the Present Embodiment>

FIG. 32 is a diagram for explaining effects of the present embodimentand shows examples of a plurality of variations of album created in thepresent embodiment. To explain in detail, FIG. 32 shows a case where thealbums 1 and 2 are created as albums for the objects A and B and themain object in the album 1 is set to the object A and the main object inthe album 2 is set to the object B.

In the present embodiment, as regards the variations of album to becreated, the total number of double-page spreads does not changedepending on the album and is the same at all times. In the example inFIG. 32, the total number of double-page spreads is four irrespective ofthe album.

Further, in the present embodiment, the number of individual double-pagespreads does not change depending on the album and is the same at alltimes. In the example in FIG. 32, the number of individual double-pagespreads of the album 1 and the album 2 is one. In each album, theindividual double-page spreads are not arranged in chronological order.

Further, in the present embodiment, the number of common double-pagespreads does not change depending on the album and is the same at alltimes. In the example in FIG. 32, the number of common double-pagespreads of the album 1 and the album 2 is three.

Further, in the present embodiment, after the common double-page spreadsare sorted in a time series, the position of the individual double-pagespread is determined. For example, all the individual double-pagespreads are inserted between individual double-page spreads where theimage capturing time difference is the maximum. Further, in anotherexample, the individual double-page spreads are inserted at a positionwhere the common double-page spreads are not divided, such as theposition before all the common double-page spreads and the positionafter all the common double-page spreads. By performing processing inthe manner like this, it is made possible to make the same the positionof the individual double-page spread of the album in all the albums, andtherefore, it is possible to cause the plurality of albums to have asense of uniformity. FIG. 32 shows the example in which the individualdouble-page spread is inserted at the position where the commondouble-page spreads are not divided (after the last common double-pagespread).

Fifth Embodiment

In the present embodiment, a double-page spread common to at least twoalbums and an individual double-page spread in each album are created.The total number of double-page spreads is the same in all the albums inaccordance with the user specification, but each of the number of commondouble-page spreads and the number of individual double-page spreads ofeach album changes in accordance with the image group and the setting.Further, the common double-page spreads and the images that are arrangedwithin the double-page spread are arranged in order of the imagecapturing time and the position of the individual double-page spread ismade the same in all the albums. In the following explanation,explanation of the portions in common to those of the first to fourthembodiments is omitted and portions different from the first to fourthembodiments are explained mainly.

<Automatic Layout Processing>

In the following, the automatic layout processing in the presentembodiment is explained by using FIG. 25A, FIG. 25B, and FIG. 26A toFIG. 26C. FIG. 25A and FIG. 25B are each a block diagram showing thefunction configuration relating to the automatic layout processing inthe present embodiment. FIG. 26A to FIG. 26C are each a flowchart forexplaining the automatic layout processing in the present embodiment.The processing that is indicated by the same symbol as that of theembodiment described previously is the same processing as that of theembodiment described previously, and therefore, explanation is omittedhere.

At S2601, a double-page spread creation unit 2501 creates a double-pagespread of the album by using the parameters generated at S510. Here,double-page spread creation processing at this step is explained byusing FIG. 26C.

At S260101, a double-page spread allocation unit 250101 divides thecandidate images acquired at S51102 into sub image groups correspondingto the number of double-page spreads set at S51103 and allocates the subimage groups. In the present embodiment, in a case where the double-pagespread creation parameter set at the immediately previous S51101 is thecommon parameter, the candidate images are divided based on the imagecapturing time acquired at S50301. On the other hand, in a case wherethe double-page spread creation parameter set at the immediatelyprevious S51101 is the individual parameter, the candidate images aredivided based on the score given at S508.

Specifically, in a case where the double-page spread creation parameterset at the immediately previous S51101 is the common parameter, thecandidate images are arranged in order of the image capturing timeacquired at S50301 and the candidate images are divided at the positionwhere the time difference between adjacent images is large. The divisionprocessing such as this is performed until the candidate images aredivided into sub image groups corresponding to the number of double-pagespreads set at S51103. That is, the division is performed (the number ofdouble-page spreads−1) times. Due to this, it is possible to create analbum in which the images are arranged in order of the image capturingtime. As will be described later, within one double-page spread, theimages do not need to be arranged in order of the image capturing time.

Further, in a case where the double-page spread creation parameter setat the immediately previous S51101 is the individual parameter, thecandidate images are allocated to each double-page spread in order fromthe candidate image whose score is the highest. For example, in a casewhere the number of double-page spreads is three, the candidate imagesare allocated to double-page spread 1, double-page spread 2, double-pagespread 3, double-page spread 1, double-page spread 2, . . . , and so on,in order from the candidate image whose score is the highest. Due tothis, as a result of the images whose score is high being allocatedequally to each double-page spread, it is possible to arrange the imagessuitable to the main objects in a balanced manner, not in a time series.

Explanation is returned to FIG. 26A. At S2602, a double-page spreadcombination unit 2502 determines the double-page spread number withinthe album for each double-page spread created by the double-page spreadcreation unit 2501 and creates the layout of the entire album. In thepresent embodiment, the common double-page spreads created based on theimage capturing time are arranged in order of the image capturing time.On the other hand, the individual double-page spread created based onthe score is inserted at the position where the time difference betweencommon double-page spreads is large.

In the following, double-page spread combination processing at S2602 isexplained by using FIG. 27 and FIG. 28. FIG. 27 is a block diagramshowing a detailed configuration of the double-page spread combinationunit 2502. FIG. 28 is a flowchart of processing performed by thedouble-page spread combination unit 2502.

At S260201, a sort unit 250201 sorts the common double-page spreadsamong the double-page spreads acquired by the double-page spreadacquisition unit 1201 based on the sort reference value found by thesort reference value calculation unit 1202. In the present embodiment,the common double-page spreads are arranged in ascending order from thecommon double-page spread whose image capturing time is the earliest.

At S2804, a number of double-page spreads adjustment unit 250202determines whether the number of double-page spreads of the album is thesame as the total number of double-page spreads set at S501. In a casewhere determination results at this step are affirmative, the processingadvances to S1306. On the other hand, in a case where determinationresults at this step are negative, that is, in a case where the numberof double-page spreads of the album exceeds the total number ofdouble-page spreads as a result of the number of double-page spreads ofeach album being set to a number larger than or equal to the totalnumber of double-page spreads, the processing advances to S260202.

At S260202, the number of double-page spreads adjustment unit 250202performs adjustment so that the number of double-page spreads of thealbum becomes close to the total number of double-page spreads set atS501 by deleting an unnecessary double-page spread among the double-pagespreads acquired by the double-page spread acquisition unit 1201. In thepresent embodiment, the number of double-page spreads is adjusted bydeleting the common double-page spread. Specifically, first, from amongthe common double-page spreads within the album, the common double-pagespread is selected, whose image capturing time of the double-page spreadcalculated by the sort reference value calculation unit 1202 is theclosest to that of the adjacent common double-page spread. Next, ofthese two adjacent common double-page spreads, the double-page spreadwhose total of the scores of the images arranged in the commondouble-page spread is smaller is deleted. After the deletion, theprocessing advances to S2804. Due to this, the common double-page spreadis deleted in order from the common double-page spread whose time isclose to that of the adjacent common double-page spread (that is, thepossibility that the two common double-page spreads represent the samescene is strong). Here, the case is explained where the commondouble-page spread to be deleted is selected based on the imagecapturing time, but the selection method of the common double-pagespread to be deleted is not limited to this. It may also be possible toselect the common double-page spread whose score of the image is smalleror adjust the total number of double-page spreads by deleting theindividual double-page spread.

At S260203, an individual double-page spread insertion unit 250203determines the double-page spread number of the individual double-pagespread among the double-page spreads acquired by the double-page spreadacquisition unit 1201. In the present embodiment, the double-page spreadnumber is determined so that all the individual double-page spreads areinserted at the position where the image capturing time differencebetween common double-page spreads is the maximum for the commondouble-page spreads sorted at S260201. The time difference between theNth common double-page spread and the (N+1)th common double-page spreadis calculated by using equation (13). In the present embodiment, all theindividual double-page spreads are inserted between the commondouble-page spreads where the time difference is the maximum, but theindividual double-page spread insertion method is not limited to this.For example, it may also be possible to insert the individualdouble-page spreads at the position where the common double-page spreadsare not divided irrespective of the time difference, such as theposition before all the common double-page spreads and the positionafter all the common double-page spreads. Further, in a case where allthe time differences between common double-page spreads are smaller thanthe predetermined value, it may also be possible to insert theindividual double-page spreads at a position where the commondouble-page spreads are not divided, such as the position before all thecommon double-page spreads and the position after all the commondouble-page spreads.

At S1306, the double-page spread acquisition unit 1201 determineswhether the processing at S1301 to S260203 has been completed for allthe albums. In a case where determination results at this step areaffirmative, the processing advances to S513 and on the other hand, in acase where the determination results are negative, the processingreturns to S1301. Due to this, it is possible to arrange the individualdouble-page spread at the position where the same double-page spreadnumber starts in all the albums. The above is the contents of theautomatic layout processing in the present embodiment. In the presentembodiment, the processing is performed in units of spread pages, butthe processing may be performed in units of pages. That is, it may alsobe possible to create the “individual page” and the “common page” inunits of double-page spreads (that is, two pages) or create the“individual page” and the “common page” in units of pages.

<Effects of the Present Embodiment>

FIG. 33 is a diagram for explaining effects of the present embodimentand shows examples of a plurality of variations of album created in thepresent embodiment. To explain in detail, FIG. 33 shows a case where thealbums 1 to 3 are created as albums for the objects A to C and the mainobject in the album 1 is set to the object A, the main object in thealbum 2 is set to the object B, and the main object in the album 3 isset to the object C, respectively.

In the present embodiment, as regards the variations of album to becreated, the total number of double-page spreads does not changedepending on the album and is the same at all times. In the example inFIG. 33, the total number of double-page spreads is four irrespective ofthe album.

Further, in the present embodiment, it is permitted for the number ofindividual double-page spreads to change depending on the album. In theexample in FIG. 33, the number of individual double-page spreads of thealbum 1 and the album 3 is one and on the other hand, the number ofindividual double-page spreads of the album 2 is two.

Further, in the present embodiment, it is permitted for the number ofcommon double-page spreads to change depending on the album. In theexample in FIG. 33, the number of common double-page spreads of thealbum 1 and the album 3 is three and on the other hand, the number ofcommon double-page spreads of the album 2 is two.

Further, in the present embodiment, after the common double-page spreadsare sorted in a time series, the position of the individual double-pagespread is determined. For example, all the individual double-pagespreads are inserted between individual double-page spreads where theimage capturing time difference is the maximum. Further, in anotherexample, the individual double-page spreads are inserted at a positionwhere the common double-page spreads are not divided, such as theposition before all the common double-page spreads and the positionafter all the common double-page spreads. By performing processing inthe manner like this, it is made possible to make the same the positionof the individual double-page spread of the album in all the albums, andtherefore, it is possible to cause the plurality of albums to have asense of uniformity. FIG. 33 shows the example in which the individualdouble-page spread is inserted at the position where the commondouble-page spreads are not divided (after the last common double-pagespread).

Sixth Embodiment

In the present embodiment, a double-page spread common to at least twoalbums and an individual double-page spread in each album are created.The image group is divided in units of scenes and for each scene,whether to create a common double-page spread or to create an individualdouble-page spread is determined. In the following explanation,explanation of the portions in common to those of the first to fourthembodiments is omitted and portions different from the first to fourthembodiments are explained mainly.

<Automatic Layout Processing>

In the following, the automatic layout processing in the presentembodiment is explained by using FIG. 34A, FIG. 34B, and FIG. 35A toFIG. 35C. FIG. 34A and FIG. 34B are each a block diagram showing thefunction configuration relating to the automatic layout processing inthe present embodiment. FIG. 35A to FIG. 35C are each a flowchart forexplaining the automatic layout processing in the present embodiment.The processing that is indicated by the same symbol as that of theembodiment described previously (see FIG. 5A to FIG. 5C) is the sameprocessing as that of the embodiment described previously, andtherefore, explanation is omitted here.

At S3501, a scene parameter setting unit 3402 exclusively sets a scenethat is applied to the double-page spread creation parameter generatedby the double-page spread creation parameter generation unit 209 basedon the scene classification results of the scene classification unit207. That is, the scene parameter setting unit 3402 creates thedouble-page spread with one parameter per scene. For example, in FIG. 6,the common parameter is applied to the sub image groups (1) to (6) andthe individual parameter is applied to the sub image groups (7) and (8).In the present embodiment, which of the common parameter and theindividual parameter is applied is determined for each scene by usingequation (14) and equation (15) below.

$\begin{matrix}{{{}_{}^{}{}_{}^{}} = {\sum\limits_{I \in S}\mspace{14mu}{{}_{}^{}{}_{}^{}}}} & (14) \\{{{}_{}^{}{}_{}^{}} = \left\{ \begin{matrix}1 & \left( {{{}_{}^{}{}_{}^{}} > {THR}} \right) \\0 & {\mspace{56mu}({otherwise})}\end{matrix}\; \right.} & (15)\end{matrix}$

Here, ^(I)T_(p) indicates the state of the score in the parameter P forthe image I, ^(I)score_(p) indicates the score of the parameter P forthe image I, S indicates the image group included in the target scene,and THR indicates a threshold value. Further, ^(S)Q_(p) is an indicatorindicating to which extent the parameter P is suitable to the scene S.By equation (14) and equation (15), the double-page spread of the sceneS is created with the parameter that is larger, that is, the indicator^(S)Q_(Pc) with the common parameter P_(c) or the indicator ^(S)Q_(Pd)with the individual parameter P_(d), which is larger. It is assumed thata score ^(I∈S)score_(p) of the image I in the scene S to which theparameter P is not applied is taken to be 0. Due to this, it is possibleto create the double-page spread with the parameter that adapts to theimage in the scene. In the present embodiment, as the indicator of theparameter suitable to the scene, the score of each image is used, butthis is not limited. For example, it may also be possible to use imageanalysis information, such as whether or not the main object is capturedand the number of captured faces, or use statistical information, suchas the number of images of each scene. Further, it may also be possibleto use the scene classification obtained at S506. For example, in thealbum of a wedding, the scenes are classified into wedding and receptionby scene classification. Then, to the scene determined to be a wedding,the common parameter setting the groom and bride as the main objects isapplied and to the scene determined to be a reception, the individualparameter setting relatives or friends as the main objects is applied.As described above, it may also be possible to incorporate in advancethe combination of the scene classification and the parameter to be usedin the application.

At S3502, a double-page spread creation unit 3401 creates a double-pagespread of the album by using the parameter that is output from the sceneparameter setting unit 3402. FIG. 34B is a detailed block diagram of thedouble-page spread creation unit 3401 according to the presentembodiment. FIG. 35C is a detailed flowchart of the double-page spreadcreation processing at S3502.

At S350201, a double-page spread candidate image acquisition unit 340101acquires candidate images that are used for the double-page spread fromamong all the image groups acquired by the image acquisition unit 203based on the score given at S508 and the double-page spread creationparameter set at S51101. In the present embodiment, the image whosescore explained at S510 is higher than or equal to a predeterminedthreshold value is acquired as the candidate image. In a case wherethere is no candidate image, the image whose score is the highest istaken as the candidate image. Here, the candidate image is acquired fromamong all the image groups, but this is not limited. It may also bepossible to exclude the image already selected by image selectionprocessing, to be described later. The above is the contents of theautomatic layout processing in the present embodiment. In the presentembodiment, the processing is performed in units of spread pages, butthe processing may be performed in units of pages. That is, it may alsobe possible to create the “individual page” and the “common page” inunits of double-page spreads (that is, two pages) or create the“individual page” and the “common page” in units of pages.

<Effects of the Present Embodiment>

According to the present embodiment, it is made possible toautomatically determine whether to apply the common double-page spreadcreation parameter or to apply the individual double-page spreadcreation parameter for each scene.

Seventh Embodiment

In the present embodiment, a double-page spread common to at least twoalbums and an individual double-page spread in each album are created.The image group is divided in unit of scenes and in a case where thedouble-page spread is created by using the double-page spread creationparameter different in the same scene, the double-page spread is createdby using the double-page spread creation parameter with which the scorebecomes higher. In the following explanation, explanation of theportions in common to those of the first to fourth embodiments isomitted and portions different from the first to fourth embodiments areexplained mainly.

<Automatic Layout Processing>

In the following, the automatic layout processing in the presentembodiment is explained by using FIG. 36A, FIG. 36B, and FIG. 37A toFIG. 37C. FIG. 36A and FIG. 36B are each a block diagram showing thefunction configuration relating to the automatic layout processing inthe present embodiment. FIG. 37A to FIG. 37C are each a flowchart forexplaining the automatic layout processing in the present embodiment.The processing that is indicated by the same symbol as that of theembodiment described previously (see FIG. 5A to FIG. 5C) is the sameprocessing as that of the embodiment described previously, andtherefore, explanation is omitted here.

At S3701, a scene parameter setting unit 3602 sets a scene that isapplied to the double-page spread creation parameter generated by thedouble-page spread creation parameter generation unit 209 based on thescene classification results of the scene classification unit 207. Forexample, in FIG. 6, the common parameter is applied to the sub imagegroups (1) to (8) and the individual parameter is applied to the subimage groups (2) and (5). Due to this, it is possible to create thedouble-page spread by using a plurality of parameters for one scene.Specifically, in the present embodiment, the common parameter is takenas the parameter that is applied to all the scenes. As regards theindividual parameter P_(d), only the scene S whose ^(S)Q_(Pd) obtainedby equation (14) and equation (15) is larger than or equal to thethreshold value THR is taken as the scene to which the individualparameter Pa is applied. It is assumed that the score ^(I∈S)score_(p) ofthe image I in the scene S to which the parameter P is not applied istaken to be 0. Due to this, it is possible to set whether to apply theindividual parameter in accordance with the image in the scene. In thepresent embodiment, as the indicator of the parameter suitable to thescene, the score of each image is used, but this is not limited. Forexample, it may also be possible to use image analysis information, suchas whether or not the main object is captured and the number of capturedfaces, or use statistical information, such as the number of images ofeach scene.

At S3702, a double-page spread creation unit 3601 creates a double-pagespread of the album by using the parameter that is output from the sceneparameter setting unit 3602. FIG. 36B is a detailed block diagram of thedouble-page spread creation unit 3601 according to the presentembodiment. FIG. 37C is a detailed flowchart of the double-page spreadcreation processing at S3702.

At S370201, a scene type determination unit 360101 determines whichdouble-page spread creation parameter is used in each scene based on thedouble-page spread creation parameter group that is output from thescene parameter setting unit 3602.

At S370202, a double-page spread candidate image acquisition unit 360102acquires candidate images that are used for the double-page spread fromamong all the image groups acquired by the image acquisition unit 203based on the score given at S508 and the double-page spread creationparameter set at S51101. In the present embodiment, the candidate imageis selected in accordance with equation (16) below among the scoresexplained at S510.

$\begin{matrix}{{{}_{}^{}{}_{}^{}} = \left\{ \begin{matrix}1 & \left( {{{{{{{I \in S_{P}}\&}\mspace{14mu}{{}_{}^{}{}_{p = P}^{}}} \geq {THR}}\&}\mspace{14mu}{{}_{}^{}{{scor}e}_{p \neq P}^{}}} < {{}_{}^{}{}_{p = P}^{}}} \right) \\0 & ({otherwise})\end{matrix} \right.} & (16)\end{matrix}$

Here, ^(I)C_(p) indicates whether the image I is a candidate image in acase of the double-page spread creation parameter P and 1 indicates scandidate image and 0 indicates a non-candidate image. S is a set ofimages in the scene used with the double-page spread creation parameterP. Each of ^(I)score_(p=p) and ^(I)score_(p≠p) indicates the score ofthe image I in a case of the double-page spread creation parameter P andthe score of the image I in a case of a parameter other than thedouble-page spread creation parameter P and THR indicates a thresholdvalue. Due to this, it is possible to select the image whose scorebecomes high only with the double-page spread creation parameter P froma specific scene, and therefore, it is possible to use the image withthe parameter with which the score becomes higher in a case where thedouble-page spread is created by using the parameter different in thesame scene. The above is the contents of the automatic layout processingin the present embodiment. In the present embodiment, the processing isperformed in units of spread pages, but the processing may be performedin units of pages. That is, it may also be possible to create the“individual page” and the “common page” in units of double-page spreads(that is, two pages) or create the “individual page” and the “commonpage” in units of pages.

<Effects of the Present Embodiment>

According to the present embodiment, in a case where the double-pagespread is created by using the double-page spread creation parameterdifferent in the same scene, it is made possible to apply a double-pagespread creation parameter with which the score becomes higher.

Eighth Embodiment

In the present embodiment, a double-page spread common to at least twoalbums and an individual double-page spread in each album are created.For example, in accordance with an image distribution of images in whichthe main object is captured, whether to change the position of theindividual double-page spread for each album or to make the same theposition of the individual double-page spread in all the albums isdetermined. In the following explanation, explanation of the portions incommon to those of the first to fourth embodiments is omitted andportions different from the first to fourth embodiments are explainedmainly.

<Automatic Layout Processing>

In the following, the automatic layout processing in the presentembodiment is explained by using FIG. 38 and FIG. 39. FIG. 38 is a blockdiagram showing the function configuration relating to the automaticlayout processing in the present embodiment. FIG. 39 is a flowchart forexplaining the automatic layout processing in the present embodiment.The processing that is indicated by the same symbol as that of theembodiment described previously (see FIG. 5A to FIG. 5C) is the sameprocessing as that of the embodiment described previously, andtherefore, explanation is omitted here.

At S3901, a double-page spread creation switching unit 3801 switchesbetween the double-page spread creation unit 210 and the double-pagespread creation unit 2101 by which the double-page spread is to becreated by using the parameter created by the double-page spreadcreation parameter generation unit 209. That is, the double-page spreadcreation switching unit 3801 switches between determining the positionof the individual double-page spread for each album by the double-pagespread creation unit 210 and making the same the position of theindividual double-page spread in all the albums by the double-pagespread creation unit 2101. In the present embodiment, as a reference ofdetermination of switching, the image distribution of images in whichthe main object is captured is used. Specifically, for the individualparameter P_(d), the indicator ^(S)Q_(Pd) is obtained by equation (14)and equation (15). In a case where the one or more scenes S whoseindicator ^(S)Q_(Pd) exceeds the threshold value THR exist for each ofall the individual parameters (that is, in a case where the imagedistribution is dense), the processing advances to S511. In the othercase, the processing advances to S2201. Here, it may also be possible toincorporate in advance the threshold value THR in the application or todetermine the threshold value THR dynamically. For example, it may alsobe possible to set a threshold value with which the top ten images whosescore is high are extracted or to set the threshold value THR differentfor each individual parameter. Due to this, in a case where the imagessuitable to the individual double-page spread and whose score is highconcentrate in a certain scene (in the present embodiment, the imagecapturing times are close), the position of the individual double-pagespread is arranged along a time series for each album in accordance withS511 and S512. On the other hand, in a case where the images suitable tothe individual double-page spread are distributed in different scenes,the position of the individual double-page spread is made the same andarranged irrespective of the time series in accordance with S2201 andS2202.

In the present embodiment, as the reference of switching between thedouble-page spread creation units 210 and 2101, the image distributionof images in which the main object is captured is used, but this is notlimited. For example, it may also be possible to make the same theposition of the individual double-page spread in a case where the numberof albums to be created is large (five or more albums, and the like) andit may also be possible not to make the same the position of theindividual double-page spread in a case where the number of albums issmall, and it may also be possible for a user to specify this by thealbum creation condition setting unit 202. The above is the contents ofthe automatic layout processing in the present embodiment. In thepresent embodiment, the processing is performed in units of spreadpages, but the processing may be performed in units of pages. That is,it may also be possible to create the “individual page” and the “commonpage” in units of double-page spreads (that is, two pages) or create the“individual page” and the “common page” in units of pages.

<Effects of the Present Embodiment>

FIG. 40 is a diagram for explaining effects of the present embodimentand shows examples of a plurality of variations of album created in thepresent embodiment. To explain in detail, FIG. 40 shows a case where thealbums 1 and 2 are created as albums for the objects A and B and theenhancement-target main object in the album 1 is set to the object A andthe main object in the album 2 is set to the object B.

As regards the variations of album to be created in the presentembodiment, the total number of double-page spreads does not changedepending on the album and is the same at all times. In the example inFIG. 40, the total number of double-page spreads is four irrespective ofthe album.

Further, the number of individual double-page spreads does not changedepending on the album and is the same at all times. In the example inFIG. 40, the number of individual double-page spreads of the album 1 andthe album 2 is one. Further, in each album, the position of theindividual double-page spread is arranged along a time series.Consequently, as shown schematically, the position of the individualdouble-page spread differs for each album.

The number of common double-page spreads does not change depending onthe album and is the same at all times. In the example in FIG. 40, thenumber of common double-page spreads of the album 1 and the album 2 isthree.

On the other hand, as shown in FIG. 32, it is also possible to make thesame the position of the individual double-page spread irrespective ofthe time series in each album.

According to the present embodiment, it is made possible toautomatically determine whether to change the position of the individualdouble-page spread for each album or to make the same in all the albums.

Other Embodiments

Embodiment(s) of the present disclosure can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

It may also be possible to appropriately combine the configuration ofeach embodiment described previously.

According to the present disclosure, it is made possible to create albumdata having both the contents common to a plurality of pieces of albumdata and the contents in accordance with each individual (object).

While the present disclosure has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2019-36977, filed Feb. 28, 2019, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An apparatus that generates a plurality of albumsincluding a first album and a second album, the apparatus comprising: atleast one processor causing the apparatus to act as a plurality of unitscomprising: (1) a page generation unit configured to generate (a) commonpages that are to be arranged in both of the first album and the secondalbum and (b) individual pages including (i) a first individual pagethat is to be arranged in the first album and is not to be arranged inthe second album and (ii) a second individual page that is to bearranged in the second album and is not to be arranged in the firstalbum, wherein the first album is album data in which (i) a first objectis set as a main object and (ii) an image including the first object isarranged in the first individual page, and wherein the second album isalbum data in which (i) a second object is set as a main object and (ii)an image including the second object is arranged in the secondindividual page; and (2) a determination unit configured to determinepositions for arranging the individual pages, wherein, in a case ofgenerating the first album, the common pages are sorted based on animage capturing time for each of the common pages, and then thedetermination unit determines a position for arranging the firstindividual page in the first album.
 2. The apparatus according to claim1, wherein the position of the first individual page arranged in thefirst album and a position of the second individual page arranged in thesecond album are the same.
 3. The apparatus according to claim 1,wherein the at least one processor further causes the apparatus to actas: an acquisition unit configured to acquire candidate images used inthe album data; an analysis unit configured to analyze the candidateimages; a specification unit configured to specify the main object ofeach album; and a setting unit configured to set a reference of pagecreation of each album, wherein the page generation unit generates thecommon pages and the individual pages by arranging the candidate imageson pages based on results of analysis by the analysis unit and thereference.
 4. The apparatus according to claim 1, wherein the imagecapturing time for each of the common pages is an average imagecapturing time of images arranged on that common page.
 5. The apparatusaccording to claim 1, wherein the determination unit determines theposition for arranging the first individual page based on a differencein the image capturing time between the sorted common pages.
 6. Theapparatus according to claim 5, wherein the page generation unitgenerates a plurality of the first individual pages, and wherein thedetermination unit determines the positions for arranging the firstindividual pages so that the first individual pages are inserted one byone in accordance with order of magnitude of the difference in the imagecapturing time between the common pages.
 7. The apparatus according toclaim 1, wherein the determination unit determines the position forarranging the first individual page so that the first individual page isinserted before or after the common pages.
 8. The apparatus according toclaim 1, wherein the determination unit adjusts a total number of commonpages and individual pages so as to be the same as a predeterminednumber of pages.
 9. The apparatus according to claim 1, wherein thedetermination unit determines the positions of the individual pages inaccordance with an image distribution of images in which the main objectis captured, a number of albums to be generated, or a specification of auser.
 10. The apparatus according to claim 1, wherein the page is aspread page.
 11. An apparatus that generates a plurality of albumsincluding a first album and a second album, the apparatus comprising: atleast one processor causing the apparatus to act as a plurality of unitscomprising: (1) a page generation unit configured to generate (a) commonpages that are to be arranged in both of the first album and the secondalbum and (b) individual pages including (i) a first individual pagethat is to be arranged in the first album and is not to be arranged inthe second album and (ii) a second individual page that is to bearranged in the second album and is not to be arranged in the firstalbum, wherein the first album is album data in which (i) a first objectis set as a main object and (ii) an image including the first object isarranged in the first individual page, and wherein the second album isalbum data in which (i) a second object is set as a main object and (ii)an image including the second object is arranged in the secondindividual page; and (2) a determination unit configured to determine aposition for arranging the individual pages, wherein a position of thefirst individual page arranged in the first album and a position of thesecond individual page arranged in the second album are the same.
 12. Amethod of generating album data a plurality of albums including a firstalbum and a second album, the method comprising the steps of: generating(a) common pages that are to be arranged in both of the first album andthe second album and (b) individual pages including (i) a firstindividual page that is to be arranged in the first album and is not tobe arranged in the second album and (ii) a second individual page thatis to be arranged in the second album and is not to be arranged in thefirst album, wherein the first album is album data in which (i) a firstobject is set as a main object and (ii) an image including the firstobject is arranged in the first individual page, and wherein the secondalbum is album data in which (i) a second object is set as a main objectand (ii) an image including the second object is arranged in the secondindividual page; and determining positions for arranging the individualpages, wherein, in a case of generating the first album, the commonpages are sorted based on an image capturing time for each of the commonpages, and then the determining determines a position for arranging thefirst individual page in the first album.
 13. The method according toclaim 12, wherein the position of the first individual page arranged inthe first album and a position of the second individual page arranged inthe second album are the same.
 14. The method according to claim 12,further comprising: acquiring candidate images used in the album data;analyzing the candidate images; specifying the main object of eachalbum; and setting a reference of page creation of each album, whereinthe generating is generating the common pages and the individual pagesby arranging the candidate images on pages based on results of theanalyzing of the candidate images and the reference of page creation.15. The method according to claim 12, wherein the image capturing timefor each of the common pages is an average image capturing time ofimages arranged on that common page.
 16. The method according to claim12, wherein the determining is comprises determining the positions forarranging the first individual page based on a difference in the imagecapturing time between the sorted common pages.
 17. The method accordingto claim 16, wherein the generating comprises generating a plurality ofthe first individual pages, and wherein the determining comprisesdetermining the positions for arranging the first individual pages sothat the first individual pages are inserted one by one in accordancewith order of magnitude of the difference in the image capturing timebetween the common pages.
 18. The method according to claim 12, whereinthe determining comprises determining the position for arranging thefirst the individual page so that the first individual page is insertedbefore or after the common pages.
 19. The method according to claim 12,wherein the determining includes adjusting a total number of commonpages and individual pages so as to be the same as a predeterminednumber of pages.
 20. The method according to claim 12, wherein thedetermining comprises determining the positions of the individual pagesin accordance with an image distribution of images in which the mainobject is captured, a number of albums to be generated, or aspecification of a user.
 21. The method according to claim 12, whereinthe page is a spread page.
 22. The apparatus according to claim 11,wherein the determination unit determines the position for arranging thefirst individual page based on a difference in the image capturing timebetween the common pages sorted by an image capturing time for each ofthe common pages.
 23. A method of generating a plurality of albumsincluding a first album and a second album, the method comprising thesteps of: generating (a) common pages that are to be arranged in both ofthe first album and the second album and (b) individual pages including(i) a first individual page that is to be arranged in the first albumand is not to be arranged in the second album and (ii) a secondindividual page that is to be arranged in the second album and is not tobe arranged in the first album, wherein the first album is album data inwhich (i) a first object is set as a main object and (ii) an imageincluding the first object is arranged in the first individual page, andwherein the second album is album data in which (1) a second object isset as a main object and (ii) an image including the second object isarranged in the second individual page; and determining a position forarranging the individual pages, wherein a position of the firstindividual page arranged in the first album and a position of the secondindividual page arranged in the second album are the same.
 24. Themethod according to claim 23, wherein the determining determines theposition for arranging the first individual page based on a differencein the image capturing time between the common pages sorted by an imagecapturing time for each of the common pages.